Surface

Surface plugin has been designed with image recognition capabilities to capture images/regions on a screen. The captured image can be used for image search, mouse Actions on the image, image labeling and optical character recognition (OCR). 

Surface Automation is particularly useful on remote machines where we do not have any handle to the content and everything is an image or pattern. It can be useful to work on remote environments such as Citrix. 

Prerequisite: 

1. For surface plugin steps, Microsoft Visual C++ version 2015-2019 Redistributable (x64) and Microsoft Visual C++ version 2015-2019 Redistributable (x86) should be installed.
   

3. This is available at the following URL,
   
4. https://support.microsoft.com/en-in/help/2977003/the-latest-supported-visual-c-downloads
   
5. Download the following two executable,
   

2. x86: vc_redist.x86.exe
   
3. x64: vc_redist.x64.exe
   

2. Surface spy is required to build workflows with Surface plugin steps. For a description on how to use Surface Spy refer the following,
   

1. Appendix 9: Surface Spy Tutorial
   

The Surface Spy tutorial also has a Sample Workflow with Web Spy 

Additional Prerequisite for Windows Server OS:

Surface Plugins use some of Windows Media Service DLL's (ex. mfplat.dll) for image recognition. You need to enable “Desktop Experience” windows feature to get the required DLL's. Following are the steps to install the required Windows Media Service- DLL's 

1. On Windows Start menu search for ‘Turn windows features on or off’
   
2. Skip the roles screen and directly go to Feature screen
   
3. Select "Desktop Experience" under "User Interfaces and Infrastructure"
   
4. Restart the system.
   

Limitation: Any type of UI Automation with Surface plugin steps is not supported in Linux based OS.

Surface Click

Description

Surface Click step is used to perform a click action on an image on the screen.

Configurations 

No.	Field Name	Description
1	Step Name	Name of the step.  This name has to be unique in a single workflow.
2	Match Pattern	Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches.  Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).
3	Click Type	Click Type has five options as below: No Click Types Description 1 Click It performs left click action. 2 Right Click It performs right click action. 3 Double Click It performs double-click action. 4 Middle Click It performs left Middle click action. 5 Mouse Move It performs Mouse Move action.
4	Position	Choose position on the selected image to perform the click action. No Positions Description 1 Left Center Left position of the Image. 2 Right Center Right position of the Image. 3 Center Center of the Image. 4 Custom Relative position from the entire Image. It could be anywhere on the screen. It is preferable to have the relative position of the relative element to click is closer to the current position.
5	Image Index	Image index is generated by the image locator of surface spy.
9	Timeout(in seconds)	Specify a timeout in seconds to locate the image.
10	Image	View the selected image.

No.	Common Buttons
1	OK	Click OK button to accept configurations.
2	Cancel	Click CANCEL button to discard configurations.
3	Edit Image(s)	Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.

No.	Field Name	Description
1	Step Name	Name of the step.  This name has to be unique in a single workflow.
2	Match Pattern	Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches.  Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).
3	Output field	Specify a field name to hold the output Boolean True or false
4	Timeout(in seconds)	Specify a timeout in seconds to locate the image.
5	Image	View the selected image.

No.	Common Buttons
1	OK	Click OK button to accept configurations.
2	Cancel	Click CANCEL button to discard configurations.
3	Edit Image(s)	Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.

No.	Field Name	Description
1	Step Name	Name of the step.  This name has to be unique in a single workflow.
2	Match Pattern	Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches.  Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).
3	Position	Position has a default value Custom. Custom position is based on a relative position to a position on an image.
4	Image Index	Image index is generated by the image locator on surface spy.  It is identified by the image locator on surface spy.  locating the image on the screen  Image index automatically populated the Image Index and is the preferred way. You may set this manually, however locating the image on the screen automatically populates the Image Index and is the preferred way.
5	Output field	Specify a field to hold the extracted text.
6	Timeout(in seconds)	Specify a timeout in seconds to get value from the image.
7	Convert To grey scale	In case the step is not able to read characters in an image accurately you may enable Convert to grey scale checkbox to fetch better results.
8	Image	Depicts the image with respect to which we need to Get Value.

No.	Common Buttons
1	OK	Click OK button to accept configurations.
2	Cancel	Click CANCEL button to discard configurations.
3	Edit Image(s)	Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.

No.	Field Name	Description
1	Step Name	Name of the step.  This name has to be unique in a single workflow.
2	Match Pattern	Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found then it throws an error. Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches.  Retrieve All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size).
3	Position	Choose position on the selected image to Set Value action. No Positions Description 1 Left Center Left position of the Image. 2 Right Center Right position of the Image. 3 Center Center of the Image. 4 Custom Relative position from the entire Image. It could be anywhere on the screen. If the relative position of the relative element to click is closer it is preferable.
4	Image Index	Image index is generated by the image locator of surface spy.
5	Input Type	Input Type has two options as below: No Click Types Description 1 Type It types the characters one by one. 2 Set Value It sets the entire text in one go. (similar to pasting text)
6	Value
7	Timeout(in seconds)	Specify a timeout in seconds to locate the image.
8	Image	Depicts the image with respect to which we need to Set Value.

No.	Common Buttons
1	OK	Click OK button to accept configurations.
2	Cancel	Click CANCEL button to discard configurations.
3	Edit Image(s)	Click Edit Image(s) button to show the Surface Spy window. You can perform capture and locate functions and you can also change editable step properties.