System for capturing and replaying screen gestures

• US 9,535,720 B2
• Filed: 03/04/2013
• Issued: 01/03/2017
• Est. Priority Date: 11/13/2012
• Status: Active Grant

• Current Assignee: Acoustic, L.P.
• Original Assignee: International Business Machines Corporation
• Inventors: Hernandez, Omar, Singh, Sukhbir, Phadnis, Neel
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): Yu, Jian
• Application Number:
  US13/783,866
• Publication Number:
  US 20140137052A1
• Time in Patent Office: 1,401 Days
• Field of Search: 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting

Abstract

A capture system may capture client events for an application session. Some client events may contain display information associated with screen gestures. The screen gestures may be associated with any user input that changes how images are displayed during the application session. For example, the screen gestures may comprise one or more of a scroll gesture, a touch start gesture, a touch move gesture, a touch end gesture, and/or a pinch gesture. In another example the screen gesture may comprise a reorientation of a device operating in the application session. A replay system may replay the application session based on the captured client events to recreate images displayed during the application session in response to the screen gestures.

Claims

1. A computer program product for replaying a stored application session, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code readable/executable by a computer to perform a method comprising:

receiving, by the computer, client events associated with the stored application session of a client device, wherein the client events include a plurality of gesture events associated with screen gestures applied to the client device during the stored application session, and wherein each client event includes a time stamp that indicates a time that the client event occurred within the stored application session;

identifying, by the computer, a screen gesture applied to the client device during the stored application session, including;

identifying, by the computer, a first gesture event associated with an initiation of the screen gesture based on a gesture identifier and a time stamp associated with the first gesture event, wherein the screen gesture prompted manipulation of an image displayed on a display screen of the client device during the stored application session, and wherein the gesture identifier associated with the first gesture event indicates a type of the screen gesture;

identifying, by the computer, a second gesture event associated with the screen gesture based on a gesture identifier and time stamp associated with the second gesture event, wherein the gesture identifier associated with the second gesture event is the same as the gesture identifier associated with the first gesture event, and wherein the time stamp associated with the second gesture event is sequential in time to the time stamp associated with the first gesture event;

extracting, by the computer, first display information from the first gesture event in response to identifying the first gesture event, wherein the first display information includes a location, size, and orientation of the image as displayed on the display screen when the first gesture event was captured;

extracting, by the computer, second display information from the second gesture event in response to identifying the second gesture event, wherein the second display information includes a location, size, and orientation of the image after the manipulation of the image as displayed on the display screen when the second gesture event was captured;

displaying, on an output interface of the computer, a skin, wherein the skin is a size of the display screen of the client device;

displaying, on the output interface of the computer within the skin, the image as displayed on the display screen of the client device when the first gesture event was captured; and

displaying, on the output interface of the computer within the skin, the image as displayed on the display screen when the second gesture event was captured;

wherein, if the screen gesture comprises a reorientation of the client device, rotating the skin and the image to simulate the reorientation of the client device.

2. The computer program product of claim 1, wherein:

the first display information comprises;

a size of the display screen of the client device; and

a size of a viewport for a web browser displayed on the display screen when the first gesture event was captured; and

the second display information comprises;

the size of the display screen of the client device; and

a size of the viewport for the web browser displayed on the display screen when the second gesture event was captured.

3. The computer program product of claim 1, wherein the first display information comprises a pixel ratio between a number of pixels of the display screen of the client device and a number of pixels used for the image displayed on the client device when the first gesture event was captured, and wherein the second display information comprises a pixel ratio between the number of pixels of the display screen of the client device and a number of pixels for the image displayed on the client device when the second gesture event was captured.

4. The computer program product of claim 1, wherein the first display information comprises a first scroll bar location for a web browser used in the stored application session when the first gesture event was captured, and wherein the second display information comprises a second scroll bar location for the web browser when the second gesture event was captured.

5. The computer program product of claim 1, wherein the method further comprises:

identifying, by the computer, the size of the display screen of the client device used in the stored application session based on either the first display information extracted from the first gesture event or the second display information extracted from the second gesture event;

displaying, on the output interface, the image, prior to the manipulation of the image, within the skin based on the location, size, and orientation of the image when the first gesture event was captured; and

displaying, on the output interface, the image, after the manipulation of the image, within the skin based on the location, size, and orientation of the image when the second gesture event was captured.

6. The computer program product of claim 1, wherein the method further comprises displaying, on the output interface of the computer, an animation indicating a finger placement associated with the screen gesture.

7. The computer program product of claim 6, wherein the animation includes an image of a hand that represents performance of the screen gesture.

8. The computer program product of claim 6, wherein the method further comprises identifying, by the computer, the type of the screen gesture based on a gesture identifier selected from the group consisting of the gesture identifier associated with the first gesture event and the gesture identifier associated with the second gesture event, wherein the animation is based, at least in part, on the type of the screen gesture.

9. The computer program product of claim 1, wherein the method further comprises:

identifying, by the computer, a height of a toolbar, from the first display information, associated with the first gesture event, wherein displaying the image prior to the manipulation of the image includes adjusting a scale factor of the image based on the height of the toolbar associated with the first gesture event; and

identifying, by the computer, a height of a toolbar, from the second display information, associated with the second gesture event, wherein displaying the image after the manipulation of the image includes adjusting a scale factor of the image based on the height of the toolbar associated with the second gesture event.

10. An apparatus, comprising:

a memory configured to store gesture events associated with screen gestures applied to a remote device during an application session; and

a computer operatively coupled to the memory and configured to;

identify, from the gesture events, a screen gesture applied to the remote device during the application session based on a series of gesture events associated with a gesture identifier and sequential time stamp values, wherein the gesture identifier indicates a type of the screen gesture and the time stamp values indicate times that the series of gesture events occurred during the application session;

extract, from the series of gesture events, a size of a screen of the remote device;

extract, from the series of gesture events, display information associated with a first gesture event that indicates a start of the screen gesture, wherein the display information associated with the first gesture event includes a location, size, and orientation;

extract, from the series of gesture events, display information associated with a second gesture event that indicates a termination of the screen gesture, wherein the display information associated with the second gesture event includes a location, size, and orientation;

display, on an output interface of the computer, a first image representing the screen of the remote device, wherein the first image includes a skin and replicates the size of the screen extracted from the series of gesture events;

display, on the output interface, a second image within the first image, wherein the second image includes elements displayed on the screen of the remote device during the first gesture event based on the display information associated with the first gesture event extracted from the series of gesture events; and

display, on the output interface, a third image within the first image, wherein the third image includes elements displayed on the screen of the remote device during the second gesture event based on the display information associated with the second gesture event extracted from the series of gesture events;

wherein, if the first and second gesture events comprise a reorientation of the remote device, rotate the skin and the second and third images to simulate the reorientation of the remote device.

11. The apparatus of claim 10, wherein the screen comprises a touch screen and the screen gesture comprises scrolling over the touch screen.

12. The apparatus of claim 10, wherein the screen gestures comprise one or more of a scroll gesture, a touch start gesture, a touch move gesture, a touch end gesture, a pinch gesture, and a reorientation of the remote device.

13. The apparatus of claim 10, wherein:

the screen gesture was applied to a webpage displayed on the remote device during the application session, wherein the second image includes a first image of the webpage as displayed during the first gesture event, and wherein the third image includes a second image of the webpage as displayed during the second gesture event.

14. The apparatus of claim 10, wherein the computer is further configured to display an animation within the first image indicating a finger placement associated with the screen gesture.

15. A method, comprising:

monitoring, by a processing circuit, for events associated with an application session;

identifying, by the processing circuit, an initiation event associated with a screen gesture from the events based on a gesture identifier associated with the initiation event being different than a gesture identifier associated with a prior, sequential event, wherein the screen gesture prompted manipulation of an image displayed during the application session, wherein the gesture identifier associated with the initiation event indicates a type of the screen gesture, and wherein the initiation event indicates an initiation of a contact with a computer touch screen during the application session;

identifying, by the processing circuit, a terminating event associated with the screen gesture from the events based on a gesture identifier associated with the terminating event being different than a gesture identifier associated with a subsequent, sequential event, wherein the gesture identifier associated with the terminating event indicates the type of the screen gesture, and wherein the terminating event indicates a release of the contact with the computer touch screen during the application session;

capturing, by the processing circuit, display information associated with the initiation event, wherein the display information associated with the initiation event comprises a location, size and orientation of the image prior to the manipulation of the image, and wherein the display information associated with the initiation event is configured to reproduce the image displayed in the application session prior to the manipulation of the image;

capturing, by the processing circuit, display information associated with the terminating event, wherein the display information associated with the terminating event comprises a location, size, and orientation of the manipulated image, and wherein the display information associated with the terminating event is configured to reproduce the manipulated image displayed in the application session in response to the screen gesture;

displaying, by a replay system on an output interface, a skin, wherein the skin is a size of the computer touch screen;

displaying, by the replay system on the output interface within the skin, the image as displayed on the computer touch screen when the initiation event was captured; and

displaying, by the replay system on the output interface within the skin, the image as displayed on the computer touch screen when the terminating event was captured;

wherein, if the initiating and terminating events comprise a reorientation of the computer touch screen, rotating the skin and the image to simulate the reorientation of the computer touch screen.

16. The method of claim 15, wherein the screen gesture comprises a hand applied to the computer touch screen associated with the processing circuit.

17. The method of claim 15, wherein the screen gesture comprises one or more of a scroll gesture, a touch start gesture, a touch move gesture, a touch end gesture, a pinch gesture, and a reorientation of the computer touch screen associated with the processing circuit.

18. The method of claim 15, wherein:

the display information associated with the initiation event comprises;

the size of the computer touch screen associated with the processing circuit used during the application session; and

a size of a viewport for a web browser used during the application session prior to the manipulation of the image; and

the display information associated with the terminating event comprises;

the size of the computer touch screen associated with the processing circuit used during the application session; and

a size of the viewport for the web browser used during the application session after the manipulation of the image.

19. The method of claim 18, wherein the display information associated with the initiation event further comprises a pixel ratio between a number of pixels used in the computer touch screen and a number of pixels used for the viewport prior to the manipulation of the image, and wherein the display information associated with the terminating event further comprises a pixel ratio between the number of pixels used in the computer touch screen and a number of pixels used for the viewport after the manipulation of the image.

20. The method of claim 15, wherein the display information associated with the initiation event comprises a first orientation identifier for the computer touch screen that indicates orientation of the computer touch screen prior to the manipulation of the image, and wherein the display information associated with the terminating event comprises a second orientation identifier for the computer touch screen that indicates orientation of the computer touch screen after the manipulation of the image.

21. The method of claim 15, further comprising:

sending, by the processing circuit, the display information associated with the initiation event to the replay system, wherein the display information associated with the initiation event is configured to enable the replay system to reproduce the image as displayed in the application session prior to the manipulation of the image; and

sending, by the processing circuit, the display information associated with the terminating event to the replay system, wherein the display information associated with the terminating event is configured to enable the replay system to reproduce the image as displayed in the application session after the manipulation of the image.