Surface

Surface

Surface

Surface plugin has been designed with image recognition capabilities to capture images/regions on a screen. The captured image can be used for image search, mouse Actions on the image, image labeling and optical character recognition (OCR). 

Surface Automation is particularly useful on remote machines where we do not have any handle to the content and everything is an image or pattern. It can be useful to work on remote environments such as Citrix. 


Prerequisite: 

  1. For surface plugin steps, Microsoft Visual C++ version 2015-2019 Redistributable (x64) and Microsoft Visual C++ version 2015-2019 Redistributable (x86) should be installed.
  1. This is available at the following URL,
  2. https://support.microsoft.com/en-in/help/2977003/the-latest-supported-visual-c-downloads
  3. Download the following two executable,
  1. x86: vc_redist.x86.exe
  2. x64: vc_redist.x64.exe


  1. Surface spy is required to build workflows with Surface plugin steps. For a description on how to use Surface Spy refer the following,
  1. Appendix 9: Surface Spy Tutorial

The Surface Spy tutorial also has a Sample Workflow with Web Spy 



Additional Prerequisite for Windows Server OS:

Surface Plugins use some of Windows Media Service DLL's (ex. mfplat.dll) for image recognition. You need to enable “Desktop Experience” windows feature to get the required DLL's. Following are the steps to install the required Windows Media Service- DLL's 

  1. On Windows Start menu search for ‘Turn windows features on or off’
  2. Skip the roles screen and directly go to Feature screen
  3. Select "Desktop Experience" under "User Interfaces and Infrastructure"
  4. Restart the system.



Limitation: Any type of UI Automation with Surface plugin steps is not supported in Linux based OS.



Surface Click

Description

Surface Click step is used to perform a click action on an image on the screen.

Configurations 

No.

Field Name

Description

1

Step Name

Name of the step.  This name has to be unique in a single workflow.

2

Match Pattern

  1. Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error.
  2. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches. 
  3. Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).

3

Click Type

Click Type has five options as below:

No

Click Types

Description

1

Click

It performs left click action.

2

Right Click

It performs right click action.

3

Double Click

It performs double-click action.

4

Middle Click

It performs left Middle click action.

5

Mouse Move

It performs Mouse Move action.


4

Position

Choose position on the selected image to perform the click action.

No

Positions

Description

1

Left

Center Left position of the Image.

2

Right

Center Right position of the Image.

3

Center

Center of the Image.

4

Custom

Relative position from the entire Image. It could be anywhere on the screen. It is preferable to have the relative position of the relative element to click is closer to the current position.


5

Image Index

Image index is generated by the image locator of surface spy. 

9

Timeout(in seconds)

Specify a timeout in seconds to locate the image.

10

Image

View the selected image. 


No.

Common Buttons


1

OK

Click OK button to accept configurations.

2

Cancel

Click CANCEL button to discard configurations.

3

Edit Image(s)

Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.



Surface Find Image

Description

Surface Find Image step is used to find an image with on the screen. It returns a Boolean value true if image is found else it returns false.

Configurations 

No.

Field Name

Description

1

Step Name

Name of the step.  This name has to be unique in a single workflow.

2

Match Pattern

  1. Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error.
  2. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches. 
  3. Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).

3

Output field

Specify a field name to hold the output Boolean True or false 

4

Timeout(in seconds)

Specify a timeout in seconds to locate the image.

5

Image

View the selected image.


No.

Common Buttons


1

OK

Click OK button to accept configurations.

2

Cancel

Click CANCEL button to discard configurations.

3

Edit Image(s)

Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.



Surface Get Value

Description

Surface Get value step is used to get value on an image on the screen.

Configurations 

No.

Field Name

Description

1

Step Name

Name of the step.  This name has to be unique in a single workflow.

2

Match Pattern

  1. Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error.
  2. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches. 
  3. Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).

3

Position

Position has a default value Custom. Custom position is based on a relative position to a position on an image.

4

Image Index

Image index is generated by the image locator on surface spy. 

It is identified by the image locator on surface spy. 


locating the image on the screen 

Image index automatically populated the Image Index and is the preferred way.


You may set this manually, however locating the image on the screen automatically populates the Image Index and is the preferred way.

5

Output field

Specify a field to hold the extracted text. 

6

Timeout(in seconds)

Specify a timeout in seconds to get value from the image.

7

Convert To grey scale

In case the step is not able to read characters in an image accurately you may enable Convert to grey scale checkbox to fetch better results.

8

Image

Depicts the image with respect to which we need to Get Value.


No.

Common Buttons


1

OK

Click OK button to accept configurations.

2

Cancel

Click CANCEL button to discard configurations.

3

Edit Image(s)

Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.



Surface Set Value

Description

Surface Set Value step is used to set value on an image on the screen.

Configurations 

No.

Field Name

Description

1

Step Name

Name of the step.  This name has to be unique in a single workflow.

2

Match Pattern

  1. Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error.
  2. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches. 
  3. Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).

3

Position

Choose position on the selected image to Set Value action.

No

Positions

Description

1

Left

Center Left position of the Image.

2

Right

Center Right position of the Image.

3

Center

Center of the Image.

4

Custom

Relative position from the entire Image. It could be anywhere on the screen. If the relative position of the relative element to click is closer it is preferable.


4

Image Index

Image index is generated by the image locator of surface spy. 

5

Input Type

Input Type has two options as below:


No

Click Types

Description

1

Type

It types the characters one by one.

2

Set Value

It sets the entire text in one go. (similar to pasting text)


6

Value


7

Timeout(in seconds)

Specify a timeout in seconds to locate the image.

8

Image

Depicts the image with respect to which we need to Set Value.


No.

Common Buttons


1

OK

Click OK button to accept configurations.

2

Cancel

Click CANCEL button to discard configurations.

3

Edit Image(s)

Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.







      Links to better reach 

            Bot Store

             EPD