Introduction To Sikuli GUI Automation Tool (Automate Anything You See On Screen)

As always, we try to bring in new things to learn for our readers. Today, let’s explore an interesting GUI automation tool – Sikuli.

“Automate anything you see” using the Sikuli Graphical User Interface (GUI) automation tool – Complete beginner’s guide to quickly set up and start using the Sikuli Script tool with these in-depth Sikuli Tutorials.

Sikuli Automates anything you see on the screen using the image recognition method to identify GUI elements. Sikuli script allows users to automate GUI interaction by using screenshots.

List Of Tutorials In This Sikuli Series

We have divided this series into 3 parts:

Tutorial #1: How It Works, How To Create A Simple Sikuli Project.
Tutorial #2: How Sikuli Can Be Used With Selenium Web Driver To Automate Webpages.
Tutorial #3: Automating Flash Based Applications Using Sikuli Tool

Sikuli GUI Automation Tool

Let’s start with the 1st part of this series.

Sikuli is a tool that automates Graphical User Interfaces (GUI) using the “Visual Image Match” method. In Sikuli, all the web elements should be taken as an image and stored inside the project. Sikuli will trigger GUI interactions based on the image visual match, the image that we have passed as the parameter, along with all methods.

Sikuli can be very useful for automating flash objects (which do not have ID or name). It can be useful in the situation, where we have a stable GUI (i.e. GUI components not changing).

Even Window-based applications can also be automated using Sikuli. Sikuli provides a friendly Sikuli-script.jar that users can easily use together with Selenium WebDriver. We can even automate Adobe Video/Audio player and Flash Games on the website using Sikuli. With simple API, it makes coding easier.

Practical Uses

Sikuli can automate Flash Objects/Flash Websites.
It’s useful to automate the Windows application. We can automate what we see on the screen.
It provides a simple API. i.e. all methods can be accessed using screen class objects.
Integration with Selenium and all other tools is easy.
Using Sikuli, we can automate desktop applications.
Most of the automation testing tools will not support flash-object automation (E.g. Selenium). Sikuli provides extensive support for automating flash objects.
It uses a powerful “Visual Match” mechanism to automate desktop & flash objects.

Benefits

Open-source Tool.
One of the biggest advantages of Sikuli is that it can easily automate flash objects.
It makes it easy to automate Windows applications.
When you’re testing an application under development and you don’t know the ID/name of the elements, then you can go with Sikuli. Sikuli checks the appearance of the image and interacts with it accordingly if a match is found.

Prerequisites:

Before getting started, we need to download and install the following software:

Any screenshot-capturing tool (For Example, DuckCapture, or qSnap)
JDK
Eclipse (detailed steps here to install JDK and Eclipse)

Steps To Create The Sikuli Java Project

Step #1: Sikuli Download – Download Sikuli from here.

Step #2: Extract the zip file which you’ve downloaded. It will contain the Sikuli-script.jar file. Save this extracted file in your local file system.

Step #3: Open Eclipse.

Step #4: Create a Java project File -> New -> Java Project

Step #5:

Right Click on the project
Go to Build Path-> Configure Build Path
Switch to the Libraries tab
Click the “Add External Jars” button and Add Sikuli-Script.jar in the Build Path.
Click “OK”

Sikuli-script.jar will be added to your project build path. You’re done. Now you can start writing Sikuli scripts for this project.

Some Sikuli Methods

#1) Creating Object for Screen Class

The screen is a base class provided by Sikuli. We need to create an object for this screen class first, then only we can access all the methods provided by Sikuli.

Syntax:
Screen s=new Screen();

#2) Click On An Element

This method used to click on the specific image present on the screen.

Syntax:
s.click(“<<image name>>”);

For Example,
s.click(“test.png”);

#3) Right Click On An Element

This method is used to right-click on the specific image present on the screen.

Syntax:
s.rightClick(“<<image name>>”);

For Example,
s.rightClick(“test.png”);

#4) Find An Element

This method is used to find a specific element present on the screen.

Syntax:
s.find(“<<image name>>”);

For Example,
s.find(“test.png”);

#5) Double Click on An Element

This method is used to trigger a double-click event on a specific image present on the screen.

Syntax:
s.doubleClick(“<<image name>>”);

For Example,
s.doubleClick(“test.png”);

#6) Check whether an Element present on the Screen

This method is used to check whether the specified element is present on the screen.

Syntax:
s.exists(“<<image name>>”);

For Example,
s.exists(“test.png”);

#7) Type a string on a Textbox

This method is used to enter the specified text in the Text box.

Syntax:
s.type(“<<image name>>”,”String to be typed”);

For Example,
s.type(“test.png”,”HI!!”);

#8) Wheeling on a particular image

This method is used to perform wheeling action on the element image.

Syntax:
s.wheel(“<<image name>>”,<<int position>>,<<int direction>>);

For Example,
s.wheel(“test.png”,25,0);

#9) Drag and Drop a Image/Element

This method is used to drag and drop a specified image from the source position to the target position.

Syntax:
s.dragDrop(“<<source image name>>”,”<<target image name>>”);

For Example,
s.dragDrop(“test.png”,”test1.png”);

#10) Roll Hover on a particular image

This method is used to perform a roll hover event on the specified image.

Syntax:
s.hover(“<<image name>>”);

For Example,
s.hover(“test.png”);

#11) Paste Copied String

This method is used to paste text on the specified textbox.

Syntax:
s.paste(“<<image name>>”,”test”);

For Example,
s.paste(“test.png”,”test”);

Sikuli Examples

#1) YouTube Video – Pause And Play A Video

Step #1: Open a YouTube video link and Capture play and pause element images using the screen capture tool.

Pause button (Note: filename is pause.png)

Play button (Note: filename is play.png)

Copy these images inside the project.

Step #2: Create a package inside the Sikuli java project created and within that create a class named “Youtube”.

Step #3: Type the following code inside that class.

package com.test;

import org.sikuli.script.FindFailed;

import org.sikuli.script.Screen;

public class Youtube {

public static void main(String[] args) throws FindFailed, InterruptedException {

// TODO Auto-generated method stub

Screen s=new Screen();

s.find("pause.png"); //identify pause button

s.click("pause.png"); //click pause button

System.out.println("pause button clicked");

s.find("play.png"); //identify play button

s.click("play.png"); //click play button

}

Step #4: Right-click on the class select Run As -> Java Application.

#2) Open Notepad And Type Some Text

Step #1: Capture the notepad icon on the desktop on the screen.

notepad_icon.png

notepad.png

Step #2: Copy these images inside your project.

Step #3: Create a class named “NotepadExample” inside your project and type the following code.

package com.test;

import org.sikuli.script.FindFailed;

import org.sikuli.script.Screen;

public class NotepadExample {

public static void main(String[] args) throws FindFailed {

// TODO Auto-generated method stub

Screen s=new Screen();

s.click("notepad_icon.png");

s.find("notepad.png");

s.type("notepad.png","This is Nice Sikuli Tutorial!!!!");

}

Step #4: Open the screen to be tested before executing the code.
Execute this file by Right click Run As -> Java Application.

#3) Drag And Drop

Step #1: Take the screenshot of the required items on the screen, and put it inside your Sikuli project.

[Note: here, downloads icon is “source.png” and flower image is “destination.png”]

Step #2: Put these pictures inside your project.

Step #3: Create a class with the name “DragAndDrop” and write the following code.

package com.test;

import org.sikuli.script.FindFailed;

import org.sikuli.script.Screen;

public class DragAndDrop {

public static void main(String[] args) throws FindFailed, InterruptedException {

// TODO Auto-generated method stub

Screen s=new Screen();

s.find("source.png");

System.out.println("Source image found");

s.find("target.png");

System.out.println("target image found");

s.dragDrop("source.png", "target.png");

}

Step #4: Execute this script by right click Run As -> Java Application.
After the execution of this script, the download icon will be dragged and dropped on the image, indicated as a target.

Before Execution:

After Execution:

Drawbacks Of This Tool

We cannot assure you that the image match will be always accurate. Sometimes, if two or more similar images are available on the screen, Sikuli will attempt to select the wrong image.
If the image appearance varies in pixel size, it will also result in the “Find Failed ” exception.
Overhead of taking too many screenshots.
If anyone of the screenshot is missing, it will affect the execution of the program.

More resources:

Simple Sikuli example here.
Detailed Sikuli documentation here.

Conclusion

Sikuli is very much useful in automating flash objects. It can be used to automate window-based applications. It is a great tool to play with elements on a screen, based on their visuals.

About the author: This is a guest post by Anitha Eswari. She is currently working as a senior test engineer having sound knowledge of manual and automation testing and various test management tools.

Next Tutorial: In the next part of this series let’s have a deep look at creating the Sikuli maven project and how to integrate Selenium with Sikuli.

Already using this tool? Please share your experience and tips. If you want to get started but have queries let us know.

Introduction to Sikuli (GUI Automation Tool) - Sikuli Tutorial Part 1 (softwaretestinghelp.com)