System and method for effecting simultaneous control of remote computers   

Abstract: The invention describes a system and methodology for controlling multiple devices simultaneously from one control device. The control device is provided with a display having a plurality of windows, each having a visual representation related to the activity of the multiple devices. By relaying commands effected at the control device through a intermediary server, it is possible to simultaneously effect a corresponding processing of the same commands at each of the multiple devices. On effecting a command the multiple devices relay back, through the server, an image representative of the result of the processing of that command to the control device for display.

FIELD OF THE INVENTION

The present invention relates to networked computers and in particular to a system and method for effecting control of remote computers in a networked computer environment. The system more particularly relates to a methodology that can be utilised in the running or execution of the same computer application on one or more remote computers concurrently so as to enable an evaluation and testing of the performance of that program on each of the remote computers.

Within a networked architecture it is known that individual computers can communicate with one another using distinct network protocols such as TCP/IP. Using these protocols it is possible for a person running a software application on a first computer to utilise data that is stored on another computer such as a file server or the like.

It is also known in the art to provide software applications that enable a remote user to effectively take control of another computer so as to run computer applications that reside on the remote computer from their own local computer. Such applications include that provided by Microsoft™ under their Remote Desktop brand and an open source software application called VNC (Virtual Network Computing). These applications make it possible to view and fully-interact with one computer from any other computer or mobile device anywhere on the Internet. VNC software is a cross-platform application which is advantageous in that it allows remote control between different types of computer, but has the restriction that only that the end user can only control one remote machine at a time; for each end user it is a 1-1 arrangement.

AnyplaceControl™ is a product that allows the user to connect to multiple machines simultaneously. However, although the user can connect to the multiple machines, he is restricted in that he can control one machine at a time, but cannot control multiple machines by passing the same action to many machines simultaneously. Therefore, although it is different to VNC in that it enables a 1-many connection, it does not allow 1-many control.

Uses of these remote desktop software applications include system administration, IT support and helpdesk applications where the technical support provider can log into the computer that is causing the difficulty and interrogate it without having to be physically present at the computer. These systems allow several connections to the same desktop thereby enabling collaborative or shared working, but with only one computer acting as the controller at a time. The remote desktop application can also be used for training purposes, whereby the instructor can view a single pupil\'s machine or several pupils\' machines, however heretofore simultaneous control of multiple machines by the teacher is not possible.

Such control of a remote computer by a local computer can be used in the realm of software testing. Software testing involves the operation of a system or application under controlled conditions and an evaluation of the results. The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they should happen and more importantly if things happen when they should not happen. The requirements for good software testing are reasonably well defined to ensure that it is a process of quality control using group of well defined methods and evaluation criteria, together with guidelines for their use, to ensure that the software products or modules are of sufficiently high quality. The testing that is required for good software testing has to be rigorous and as such it will be apparent that it adds to the cost of development.

It can be applied to all or some of the following activities: Robustness, Reliability, and Security Testing Ensuring that the software provides fault-free service under specific hardware and software environments. Logo Compliance and Certification Testing Ensuring the application under test meets standards to compliant with a standard such as Sun “JavaVerified” or Microsoft “Designed for Windows”. Compatibility Testing Application and hardware compatibility testing for new versions of platform and system software. Interoperability/Integration Testing Ensuring that various components of the system work well together in selected scenarios. Globalisation/Localisation Testing to ensure that a product meets well defined globalisation standards.

As the application of software testing is pervasive within a computer software architecture, it would therefore be advantageous if certain portions of the test strategy could be automated. Despite the benefits in cost and time, automated testing to date has not been perfect. Some examples have been deployed in what is called user interface automated testing where one or more of the following actions are used: Capturing controls, Manipulating controls, Acquiring data from controls.

In capturing controls it is necessary to know what you are looking for. For example in an Windows™ environment there are three types of controls: 1. Standard Windows™ controls—buttons, text boxes, custom controls, 2. Custom owner drawn controls are controls that have the functionality of standard controls but also have custom functionality added by the developer. Windows provides support for “owner drawn” controls and menus by allowing the developer to override methods in the base classes of menus or dialogs and implement new appearance and behaviour of the control or menu. 3. Painted controls—controls that have the functionality of standard controls, but also have functionality added by the developer. In addition to the “Custom owner draw controls”, the painted controls take responsibility for processing windows messages, such as WM_PAINT, themselves.

Current automated testing tools come in three general categories, each with different operating modes, namely: Simple capture—record and playback, Object orientated automation, Image-based component discovery.

It will be appreciated that each of these techniques has different architecture and different characteristics. The Simple Capture Record/Playback has the advantage of being simple, while it suffers the disadvantages of being sensitive to graphical user interface changes, sensitive to application position on the screen and having playback synchronisation issues. The programmatic approach can adjust to changes in the application user interface more easily than the former approach and can handle playback synchronisation issues better. However, there is a skills issue where the tester has also to be a developer. The issue of test case script development brings with it version control and change management, much like a software development process. The third approach utilises images that are generated and utilised during the running of the program being tested. Images are taken of regions around the mouse at the time of some mouse interaction. This image is then stored. At a later time the stored image is used to identify where the mouse action should be taken either during playback of a test script or where the action should be taken on some other system.

An example of an available product that utilises image based component discovery in an automated testing context is a product provided by Redstone™ under their Eggplant™ brand. Eggplant™ is a software testing tool that utilises VNC (discussed above) which is specifically directed towards testing what a human will experience when using a software application. The Eggplant™ software operates by separating the issues of finding a region of interest and finding the area where the user input is to be applied.

Furthermore, Eggplant does not recognise objects in the sense described for Object Oriented automated testing. The Eggplant “objects” are screenshots of identifying features, they are not objects in the sense of widgets, with defined data and behaviour. The Eggplant “objects” are defined by the user, using the movable and resizable window that defines the region of interest.

There is therefore a need to provide a system and methodology that enables a control of a plurality of computing devices through the desktop of a first computing device, with desktops of the controlled devices being displayed on a display associated with the first device. There is a further need for an image based discovery testing application that can be easily rolled out over multiple computers and provides a robust and secure testing architecture.

SUMMARY

These and other needs are addressed by a system and method in accordance with the present invention that provides for an automation of testing or remote management of computers within a networked environment. A network architecture in accordance with the invention enables a 1-many connection and control between a first computing device and a plurality of other computing devices. The architecture enables this 1-many connection by provision of a client target module which is executable on one or more systems under test, each of these being defined as target devices, a client viewer module which is executable on a local computer and allows a user to view and control each of the systems under test on a local desktop—referred to as a client device and a dispatch server module which provides for an interface between each of the client viewer modules and client target modules—referred to as a server device. The dispatch server module is a server application executable on a computer and configured to mediate all interactions between the user and each of the systems under test. Interfaceable with the dispatch server module may be provided an interpretation server module which may be configured to effect one or more of a plurality of possible tasks including for example image processing, image segmentation, image interpretation, database lookup and generation of generic information, such as XML descriptions of the image being interrogated.

Using a network architecture in accordance with the invention enables a user to perform automated testing or remote management on one or more remote computers simultaneously; the interface provided between the user and the systems under test enables a connection to and control of each of the systems under test concurrently. It will be appreciated that the word computer is used in its general sense to encompass any computing device or electronic device running an operating system and that it is intended to encompass within the present invention any device such as for example personal digital assistants (PDAs), mobile telephones etc., as well as a conventional personal computer or laptop.

Accordingly the invention provides a system as detailed in claim 1 with advantageous embodiments provided in the dependent claims. The invention also provides a method in accordance with claim 23.

These and other features will be better understood with reference to the following drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows in schematic form, a network architecture in accordance with the teachings of the invention.

FIG. 2 shows an example of the processing implemented across the network architecture in accordance with the invention.

FIG. 3 shows a modification of the architecture of FIG. 1 to include an additional client viewer device.

FIGS. 4a to 4c illustrate socket communications between components of the present invention.

FIG. 5 shows a screenshot from a client device showing operation of a system in accordance with the teaching of the present invention, with two machines connected.

DETAILED DESCRIPTION

A system in accordance with the teachings of the invention will now be described with reference to the following nomenclature which is provided for ease of explanation and no inference is to be taken from the naming protocol used. As shown in FIG. 1, a system 100 in accordance with the teaching of the invention has several components: One or more Shadow Client Targets (SCT) 105 Shadow Dispatch Server (SDS) 110 coupled to a data storage device 110A Shadow Client Viewer (SCV) 115 Shadow Interpretation Server (SIS) 120, coupled to a data storage device 120A

The SIS may be considered an optional feature to the architecture as it enables further processing routines and functionality to be achieved within the 1-many connection.

In the following sections we will go through each of the individual components separately.

The SCT is a software module that is operable on the system under test or machine that the user desires to be controlled remotely. It is provided as a multi-threaded piece of software running on the system under test and performs the following functional steps: SCT opens a port at which to listen SCT creates a socket for data transfer SCT actively tries to connect with a Shadow Dispatch Server (SDS).

The SCT may be run or executed in one of two modes. The SCT can be run as a regular executable file. Alternatively, it may be installed on the operating system as a service that starts up automatically whenever the SCT machine is restarted.

If the SCT is running as a regular executable file then the SCT presents the user with a dialog box prompting the user to specify the SDS from a list of possible SDS\'s. Alternatively, if the SCT is running as an operating system service then the SCT will use configuration data in order to determine which SDS to which to connect. The configurable data is set through for example an initialization file, an “INI” file, that the user may edit.

During operation of the system of the invention, the SCT receives from the SDS packets containing data specifying command inputs such as mouse and keystroke information. The SCT generates and sends a sequence of graphic files such as bitmaps to the SDS which are screenshots of the machine in its current state. The SCT performs an action resultant from the command inputs and then provides a graphic file illustrating the result of that command input, in addition to the sequence of graphics being provided regularly, as evidence of the result of that action. In this way it will be understood that what the SCT sends are periodic regular interval screenshots of the current display at that SCT and, when that display changes as a result of some command, a further additional update showing how the display has changed.

It will be understood that a system in accordance with the present invention may be deployed across a plurality of different machines and operating systems, yet at any one particular instance that only one subset of the machines is being utilised. The SCT may connect to a single SDS which is a member of the set of available SDS\'s. A SCT may only connect to one SDS in an active mode, but may be connected to many SDS\'s concurrently.

During operation of the system, the SCT receives input data from the SDS, through the data socket. It will be understood that the communication between the individual components of the system is through network protocols such as TCP/IP. These data may comprise one or more (but not limited to) of: Keyboard input and supporting data Mouse/Pointer input and supporting data Request for information about the SCT hardware setup Request for a screenshot to be taken Request to retrieve clipboard contents Request to insert data into clipboard

The keyboard or mouse actions are sent to the operating system queue. If the input also included a request that a screenshot be taken, that is, if the SCV is in recording mode, then the SCT waits for the operating system to process the input before it takes the screenshot. FIG. 2 shows an example of such a screen shot request. In step 1, a request is generated at the SCV 115. This request is received at the SDS where the appropriate SCTs are identified (step 2). The request is then relayed to the appropriate SCTs (Step 3), where on receipt it is processed and the relevant screen shot taken (Step 4). This is compressed locally at the SCT and relayed back to the SDS (Step 5). On receipt, it is then relayed by the SDS to the SCV, still in compressed format (Step 6). On receipt at the SCV it is uncompressed and displayed in the relevant window for that SCT on the display of the SCV (step 7).

The SIS contains the widget finding functionality. If the supporting information that accompanies a user input request indicates that a widget finding routine should be used then the SIS will perform the widget finding routine, and pass the necessary information to the SCT via the SDS, which will then apply the user input in an appropriate place.

If the SCT clipboard contents are modified then a message is sent to the SDS that represents this event. The message also contains the clipboard contents and any necessary supporting information.

It will be understood that the performance of tasks that are initially effected elsewhere in the architecture is only one portion of a 1-many connection. It is also necessary for confirmation of these tasks to be returned to the controlling device, the SCV. The SCT therefore is required to output a series of data outputs corresponding to the activity performed locally on the SCT device. The SCT achieves this by taking one or more screen shots which may then be compressed using known compression libraries such as ZLIB or the like. The compressed file is sent through the data socket to the SDS. The end user, i.e. the person controlling the SCV, can configure the frame rate—i.e. the number of screenshots taken per second, which may be set to a default rate of for example 5/sec.

The SCT sends back responses based on the received request to process user input. These responses can include supporting information such as the mouse position of where the action was applied. Further, this data may include a compressed screenshot representing the screen state subsequent to the SCT applying the action.

If the SIS widget finding routine finds more than one potential result then the SIS will send back a request indicating that further supporting information is needed.

The SCT can also send back a message containing information about its hardware setup and/or its clipboard contents.

The Shadow Dispatch Server is a multi-threaded piece of software providing an interface between one or more SCTs and an SCV and performs the following functional steps: On execution it waits for connections from SCT\'s and SCV\'s. SDS instantiates a socket and thread for each SCT. SDS instantiates a socket and thread for each SCV. SDS instantiates other threads for internal tasks.

The SDS connects to a subset of the available machines, which may be many but at least 1. As mentioned above, with reference to the SCT, some deployments of the system of the present invention may involve installation on a plurality of machines & operating systems, yet at any one instant of operation, there may exist some machines that are not connected to any SDS.

The SDS has input and output communications with both SCV\'s and SCT\'s, using network protocols such as TCP/IP for data transfer. In essence it provides the interface between the SCV and the SCT. While it is possible that the SDS makes connection with many SCV\'s it is the case that only 1 of the SCVs is in full control of any one particular SCT. The other SCV\'s that are connected via the SDS to this SCT can only observe and take a “feed” of screenshots from the SDS for display purposes only.

Each SDS thread devoted to a single SCT does the following: Supports a connection to the SCT. The SCT in fact makes a connection to the SDS. Receives requests from the SCV. Based on the SCV request message content the SDS may decide to send this request to the SIS for processing. If this is the case then the current thread will wait for a response from the SIS before processing any other requests from the SCV associated with this thread. The request is forwarded to the appropriate targets. It should be noted that the same one request from a SCV may be sent to many SCT\'s. This one to many scenario occurs when the SCV is in controller mode and is controlling many machines simultaneously. Receives responses from the SCT Based on the response from the SCT the SDS may forward the response to a SIS for further processing. If the SCV is in recording mode then the SDS will save the response to its data storage device. The data is stored in electronic format in such a way that it is easily retrievable and can be used for playback purposes at a later stage. The SDS may decide to forward the SCT response to the appropriate SCV. Receives data from the SCV—in the form of XML describing mouse and keyboard movements. Relays the XML data to the SCT. Relays the compressed screenshot data to the SCV.

As an interface module, it will be appreciated that the SDS receives and transmits a plurality of inputs and outputs, both from the SCT and the SCV.

The following is a list of the types of inputs typical to the SDS: SCV request to take a screenshot of the associated SCT. SCV request for SCT hardware and software information. SCV request to process user input plus supporting information. SCV message indicating that the current remote machine is now the controller. SCV message indicating that the current remote machine\'s recording state has changed. SCV message indicating that the current remote machine\'s playback state has changed. SCV request to return a list of all connected SCT\'s. SCV request to close this connection. SCV message indicating that the clipboard content of this SCV has changed plus clipboard information. SCV request to take a manual screenshot. SIS response. SCT compressed screenshot and supporting information. SCT hardware and software information. SCT message indicating that the clipboard content of this SCT has changed plus clipboard information. SCT user input response and supporting information. SCT manual screenshot response. SCT request to close this connection.

With regard to outputs, the SDS will save recorded data to its data source-thereby providing a cache functionality. This data is stored in electronic form and can be used at a later stage for playback purposes. Typical data outputs from the SDS include: Request to SCT to take a screenshot of the associated SCT. Request to SCT for SCT hardware and software information. Request to SCT to process user input plus supporting information. Message to SCT indicating that the clipboard content of this SCV has changed plus clipboard information. Request to SCT to take a manual screenshot. Request to SIS to analyse data. SCT compressed screenshot and supporting information. SCT hardware and software information. SCT message indicating that the clipboard content of this SCT has changed plus clipboard information. SCT user input response and supporting information. SCT manual screenshot response. SCT request to close this connection.

The SCV is also a multithreaded application. The SCV connects to a single SDS, and may do so in a viewing capacity or in an active capacity. A SCV may connect to many SDSs, although only one will be in active mode. In an active capacity, the SCV controls the operation of the SCT. In a viewing capacity, an SCV simply receives display information from each connected SCT and displays each SCT—no user input is sent to by a viewing SCV to any SCT. FIG. 1 shows an example of a one group configuration where one SCV 115 is connected to one SDS 110. The SCTs 105 are all under the control of the SCV and the SCV is in active mode in this configuration. In the two group example of FIG. 3, two SCVs 115A, 1158 are connected to one SDS 110. In this configuration one of the SCVs, for example SCV 115A, is in control of all of the machines of its group. In Group 1 the SCV associated with this configuration is in control of all of the machines in Group 1. There are three SCTs that are members of both groups. In this example, the SCV associated with Group 2, SCV 1158, can only view the SCT in Group 1 and is not in control of it, but is in full control of the other SCTs in Group 2.

For our purposes, a “relevant” SCT, in the context of being connected to a SCV, is one for which the SCV has control over the SCT in an active fashion.

FIG. 4 shows the communications sockets for data to and from the SCV and SDS. It also shows the communication sockets between the SDS and each individual SCT. The SCV has a single thread for each socket connection to the SDS corresponding with a SCT. It is through these sockets that the SCV communicates with each SCT, via the SDS. FIG. 4A shows the communication sockets that are present, whereas FIG. 4B shows the data flow for sockets when a first SCT, SCT_1, is a controller and a second SCT, SCT_2, is being interacted with directly through the SCV, and not though the controller. FIG. 4C shows the data flow for the sockets when SCT_1 is a controller.

In order to effectively control each of the SCTs, it is important for the user to be cognisant of the activities on each of the target machines. The system of the invention achieves this by providing a window display on the SCV of each of the SCTs to which it is connected. An example is shown in the screen shot of FIG. 5, where it is evident that the SCV has a window for each SCT 505, 510, that has been selected. It is also possible to provide a second window 505A, 510A, a loupe window for each SCT, where the area of activity on each of the SCTs is enlarged. The mouse and keyboard information for each window is collected using standard event listeners for these types of events.

Within the SCV a controller window corresponding to a particular SCT may be specified using a menu option or a short cut key. The SDS is provided with this information by the SCV. This controller machine window in the SCV passes its events to the SDS through the data sockets. The SDS passes the events to the SCT corresponding to the controller window, but it also passes these same events to each (non-controlling) SCT that is connected to it. The SDS does not pass all of the events to all of the SCTs connected it, as some may not be relevant to the specific SCV, for example SCV 115B in FIG. 3 does not need to see activities relevant to Group 1. In this situation the user has remote control over all relevant SCTs (machines) simultaneously through effecting actions on one window alone in the SCV.

In the case that there is a controller window set and the user moves the mouse over another window in the SCV, a different situation to the above pertains. In this case event data are passed through the socket pertinent to the SCT, to the SDS and from there to the SCT. However in this case the SDS does not relay the events to all relevant (in the sense described in the previous paragraph) SCTs connected to it. In this way the user has remote control of one single machine, see FIG. 4B.

The SDS has two data sockets per (relevant) SCT. Each data socket is uni-directional and serves different purposes. One socket contains data messages such as; requests for screenshots, mouse and keyboard data, closing messages and other text based data. The other socket contains data coming from the SCT such as messages and compressed screenshots.

As discussed above, some deployments of the invention will be sufficient using only a SCV and an SDS through which the SCV interfaces with one or more SCTs. Where additional functionality is required in the interaction, the invention provides an interpretation server, the Shadow Interpretation Server (SIS) 120. The SIS connects to the SDS. It is a multithreaded application. The purpose of the SIS is to perform digital image processing and database lookup operations.

The functionality of the SIS includes the following: Image segmentation using digital image processing techniques such as, but not limited to, threshold segmentation, edge based segmentation, region growing, non-linear scale space linking and/or others. The regions into which the image is segmented contain the feature on which the user clicked. We now have a region of a certain size and boundary from which we can extract other information. Feature vector extraction—using the specified region of the image, techniques such as gray scale and colour histograms, n\'th order moments and/or wavelets are applied to get a measure that describes an image region. This measure may be expressed as a vector quantity so that mathematical techniques may be used on it. Such feature vectors may be required for several contiguous segments. Object recognition using the feature vectors. Identification of an object is governed by using fuzzy logic, neural networks, support vector machines and/or case based reasoning applied to the group of feature vectors. Reference objects may be stored in a database format against which to compare the unknown feature.

One of the functions of the SIS is, given a coordinate in an image, to find the object associated with that point. The SIS does this in the following manner: The image is segmented—using a digital image processing segmentation technique. A feature vector is extracted for the segments (regions) around the mouse, such as, but not limited to, a histogram or a moment measure. These feature vectors define the “object” in the image from which the coordinate was specified. These feature vectors are compared to reference feature vectors in order to assess similarity, using a distance metric, such as, but not limited to; the Euclidean metric, the Manhattan metric or Earthmover distance. Reference feature vectors are stored in a database. The database having been manually constructed to contain such data as might be necessary when examining a set of screenshots. Reference feature vectors are retrieved using SQL. Reference feature vectors have an identification with “objects” that might be encountered when examining a set of screenshots, such as “menu”, “toolbar”. The identification of an object is made by providing a set of candidate objects that have feature vectors most closely resembling (similarity being used in the sense defined above) those feature vectors extracted from the source image. For most images this list of candidate object identifications will be of length 1. If the list is longer than 1, then the process has not made a unique object identification. In order to make the identification unique, we will either seek user intervention to decide which is the correct identification or increase the number of feature vectors used and repeat the process until a unique identification is made.