FreshPatents.com Logo
stats FreshPatents Stats
n/a views for this patent on FreshPatents.com
Updated: April 14 2014
newTOP 200 Companies filing patents this week


    Free Services  

  • MONITOR KEYWORDS
  • Enter keywords & we'll notify you when a new patent matches your request (weekly update).

  • ORGANIZER
  • Save & organize patents so you can view them later.

  • RSS rss
  • Create custom RSS feeds. Track keywords without receiving email.

  • ARCHIVE
  • View the last few months of your Keyword emails.

  • COMPANY DIRECTORY
  • Patents sorted by company.

AdPromo(14K)

Follow us on Twitter
twitter icon@FreshPatents

Automating execution of arbitrary graphical interface applications

last patentdownload pdfdownload imgimage previewnext patent


20130019171 patent thumbnailZoom

Automating execution of arbitrary graphical interface applications


Interaction with an application interface is monitored and an image of the interface prior to a user interaction is compared with an image after the user interaction to identify a region of interaction. The comparison may be carried out by detecting that a threshold change has occurred using a computationally inexpensive algorithm and then drawing boundaries for the region using a more computationally intensive algorithm. A marker in a UI evolution timeline is associated with an image of the region and coordinates of the interaction normalized and relative to the region. An interaction interval and a profile of change are also stored in association with the marker. For playback, a script can be constructed from the timeline to control automated execution of the application by matching a current interval and profile to the interval and profile for a recorded interaction, and automatically applying that interaction to the current user interface.
Related Terms: Algorithm User Interface Coordinates Graph

USPTO Applicaton #: #20130019171 - Class: 715704 (USPTO) - 01/17/13 - Class 715 
Data Processing: Presentation Processing Of Document, Operator Interface Processing, And Screen Saver Display Processing > Operator Interface (e.g., Graphical User Interface) >Playback Of Recorded User Events (e.g., Script Or Macro Playback)

Inventors: Daniel G. Mounty, Michael B. O'connor

view organizer monitor keywords


The Patent Description & Claims data below is from USPTO Patent Application 20130019171, Automating execution of arbitrary graphical interface applications.

last patentpdficondownload pdfimage previewnext patent

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of copending U.S. patent application Ser. No. 13/180,015 filed Jul. 11, 2011.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to computer software, and more particularly to a method of characterizing user interactions with a graphical user interface (GUI) of a program application for later use in automated execution of the GUI, such as in quality assurance testing.

2. Description of the Related Art

Computers and computer software have become ubiquitous in modern society, whether in powerful systems such as servers and technical workstations, or simpler hand-held devices such as smartphones. It is often desirable to automate the execution of a software application for a variety of purposes including testing, development, demonstration, or business process automation. Automated execution of an application is typically carried out using a script prepared by a designer or tester, i.e., a series of commands at the user interface level of the application. Early computer systems and applications used rudimentary command line (text) interfaces and basic script formats, but modern systems provide a more intuitive approach with graphical user interfaces (GUIs) that can be manipulated by other means such as a special keystroke, a keystroke combination, a graphical pointing device (mouse), or a touchscreen.

The advent of GUI-based applications has greatly increased user friendliness and acceptance of software applications but has lead to a concomitant increase in the complexity of the interface. Not only has the interface become populated with a larger number of interface objects and varieties of such objects (e.g., text fields, action buttons, check boxes, radio buttons, hyperlinks, etc.) but there is also a heavier reliance on these features of the GUI. This interface revolution presents increasingly difficult challenges for automated execution of software applications, particularly in the context of quality assurance. While it is important to ensure that any software application is working as intended, it has become necessary to diligently explore the user interface in order to adequately test common features, special functionalities, logic paths, and programmatic aspects of the application.

SUMMARY

OF THE INVENTION

The present invention relates to a method of recording the evolution of a user interface for a software application, by monitoring image variations in the user interface as a user interacts with the software application executing on a computer system, detecting a user interaction with the user interface during said monitoring, comparing a first image of the user interface immediately prior to the user interaction with a second image of the user interface immediately after the user interaction to identify a region of interaction related to the user interaction, placing a marker in a user interface evolution timeline to indicate a point in time when the user interaction occurred, and storing an image of the region of interaction and coordinates of the user interaction in association with the marker. The coordinates of the user interaction are preferably normalized and relative to the region of interaction. The comparison may be carried out by detecting that a change in the user interface has occurred using a computationally inexpensive algorithm such as fuzzy pattern matching which compares the first image to the second image, and then drawing boundaries for the region of interaction using a more computationally intensive algorithm such as image diffs between the first image and the second image. If the user interaction resulted in a new user interface element which is present in the second image but not present in the first image, or the user interaction resulted in removal of a user interface element which is present in the first image but not present in the second image, the image of the region of interaction is created using the first image. An interaction interval and a profile of change for the user interaction can also be stored in association with the marker. For playback purposes, a GUI script can be constructed from the user interface evolution timeline and used to control automated execution of the software application by matching a current interaction interval and current profile to the interaction interval and profile of change for the user interaction, locating a target region in an image of the current user interface which corresponds to the region of interaction, and automatically applying the user interaction to the current user interface based on the coordinates normalized and relative to the target region.

The above as well as additional features and advantages of the present invention will become apparent in the following detailed written description.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings.

FIG. 1 is a block diagram of a computer system programmed to carry out automated GUI testing in accordance with one implementation of the present invention;

FIGS. 2A-2C are screenshots illustrating user interactions with a graphical user interface of a software application which are recorded in accordance with one implementation of the present invention;

FIG. 3 is a pictorial representation of a user interface evolution timeline having markers with associated user interaction information in accordance with one implementation of the present invention;

FIG. 4 is a chart illustrating the logical flow for creating a GUI script in the form of a user interface evolution timeline such as that illustrated in FIG. 3 in accordance with one implementation of the present invention; and

FIG. 5 is a chart illustrating the logical flow for automatic execution of a graphical interface application using a user interface evolution timeline such as that illustrated in FIG. 3 in accordance with one implementation of the present invention.

The use of the same reference symbols in different drawings indicates similar or identical items.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Download full PDF for full patent description/claims.

Advertise on FreshPatents.com - Rates & Info


You can also Monitor Keywords and Search for tracking patents relating to this Automating execution of arbitrary graphical interface applications patent application.
###
monitor keywords



Keyword Monitor How KEYWORD MONITOR works... a FREE service from FreshPatents
1. Sign up (takes 30 seconds). 2. Fill in the keywords to be monitored.
3. Each week you receive an email with patent applications related to your keywords.  
Start now! - Receive info on patent apps like Automating execution of arbitrary graphical interface applications or other areas of interest.
###


Previous Patent Application:
Automating execution of arbitrary graphical interface applications
Next Patent Application:
Labels and tooltips for context based menus
Industry Class:
Data processing: presentation processing of document
Thank you for viewing the Automating execution of arbitrary graphical interface applications patent info.
- - - Apple patents, Boeing patents, Google patents, IBM patents, Jabil patents, Coca Cola patents, Motorola patents

Results in 0.51962 seconds


Other interesting Freshpatents.com categories:
Medical: Surgery Surgery(2) Surgery(3) Drug Drug(2) Prosthesis Dentistry   -g2-0.2855
     SHARE
  
           

FreshNews promo


stats Patent Info
Application #
US 20130019171 A1
Publish Date
01/17/2013
Document #
13479002
File Date
05/23/2012
USPTO Class
715704
Other USPTO Classes
715810
International Class
06F3/048
Drawings
8


Algorithm
User Interface
Coordinates
Graph


Follow us on Twitter
twitter icon@FreshPatents