SYSTEMS, METHODS, MEANS, AND MEDIA FOR RECORDING, SEARCHING, AND OUTPUTTING DISPLAY INFORMATION

• US 20080301101A1
• Filed: 02/27/2008
• Published: 12/04/2008
• Est. Priority Date: 02/27/2007
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Laadan, Oren, Nieh, Jason, Phung, Dan, BARATTO, Ricardo, Potter, Shaya J.
• Granted Patent: US 8,214,367 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Systems for recording, searching, and outputting display information are provided. In some embodiments, systems for recording display information are provided. The systems include a virtual display that: intercepts display-changes describing changes to be made to a state of a display; sends the display-changes to a client for display; records the display-changes; and a context recorder that records context information describing a state of the display derived from a source independently of the display changes and independently of screen-images. In some embodiments, the systems further include a display system that generates an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information. In some embodiments, the virtual display further records screen-images; and the display system further generates the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

Claims

1. A system for recording display information, comprising:

a virtual display that;

intercepts display-changes describing changes to be made to a state of a display;

sends the display-changes to a client for display;

records the display-changes; and

a context recorder that records context information describing a state of the display derived from a source independently of the display changes and independently of screen-images.

2. The system of claim 1, wherein the display-changes are display commands, the context information is derived from an accessibility mechanism, and the context information includes text.

3. The system of claim 1, further comprising:

a display system that;

generates an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

4. The system of claim 3,

wherein the virtual display further records screen-images;

and

wherein the display system further generates the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

5. The system of claim 3, further comprising:

an execution recorder that records execution of an application generating the display; and

an execution revival system that revives execution from a time of the output screen-image.

6. The system of claim 1, further comprising:

a search tool that;

receives a user query from a user;

determines whether the user query matches any of the context information; and

a display system that

based on the whether the user query matches any of the context information, outputs at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

7. The system of claim 1, further comprising:

a search tool that;

receives a user query from a user;

determines whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, determines a correspondence between context information that matched the query and at least one of the display changes; and

a display system that;

generates an output screen-image based at least in part on the at least one of the display-changes determined to have a correspondence with the context information that matched the query.

8. A system for recording display information, comprising:

means for intercepting display-changes describing changes to be made to a state of a display;

means for sending the display-changes to a client for display;

means for recording the display-changes; and

means for recording context information describing a state of the display derived from a source independent of the display changes and independent of screen-images.

9. The system of claim 8, wherein the display-changes are display commands, the context information is derived from an accessibility mechanism, and the context information includes text.

10. The system of claim 8, further comprising:

means for generating an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

11. The system of claim 10, further comprising:

means for recording screen-images; and

means for generating the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

12. The system of claim 8, further comprising:

means for receiving a user query from a user;

means for determining whether the user query matches any of the context information; and

means for outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes, based on the whether the user query matches any of the context information.

13. A method for recording display information, comprising:

intercepting display-changes describing changes to be made to a state of a display;

sending the display-changes to a client for display;

recording the display-changes; and

recording context information describing a state of the display derived from a source independent of the display changes and independent of screen-images.

14. The method of claim 13, wherein the display-changes are display commands, the context information is derived from an accessibility mechanism, and the context information includes text.

15. The method of claim 13, further comprising:

generating an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

16. The method of claim 13, further comprising:

recording screen-images; and

generating the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

17. The method of claim 13, further comprising:

receiving a user query from a user;

determining whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

18. A system for recording display information, comprising:

a virtual display that;

intercepts display-changes describing changes to be made to a state of a display;

sends the display-changes to a client for display;

records the display-changes; and

a context recorder that records context information derived from an accessibility mechanism and describing a state of the display

19. The system of claim 18, wherein the display-changes are display commands, the context information is derived from an accessibility mechanism, and the context information includes text.

20. The system of claim 18, further comprising:

a display system that;

generates an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

21. The system of claim 20,

wherein the virtual display further records screen-images;

and

wherein the display system further generates the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

22. The system of claim 18, further comprising:

a search tool that;

receives a user query from a user;

determines whether the user query matches any of the context information; and

a display system that

based on the whether the user query matches any of the context information, outputs at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

23. A computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for recording display information, the method comprising:

intercepting display-changes describing changes to be made to a state of a display;

sending the display-changes to a client for display;

recording the display-changes; and

recording context information describing a state of the display derived from a source independent of the display changes and independent of screen-images.

24. The computer-readable medium of claim 23, wherein the display-changes are display commands, the context information is derived from an accessibility mechanism, and the context information includes text.

25. The computer-readable medium of claim 23, the method further comprising generating an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

26. The computer-readable medium of claim 25, the method further comprising:

recording screen-images; and

generating the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

27. The computer-readable medium of claim 23, the method further comprising:

receiving a user query from a user;

determining whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

28. A computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for recording display information, the method comprising:

intercepting display-changes describing changes to be made to a state of a display;

sending the display-changes to a client for display;

recording the display-changes; and

recording context information derived from an accessibility mechanism and describing a state of the display

29. The computer-readable medium of claim 28, wherein the display-changes are display commands and the context information includes text.

30. The computer-readable medium of claim 28, the method further comprising generating an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information.

31. The computer-readable medium of claim 30, the method further comprising:

recording screen-images; and

generating the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

32. The computer-readable medium of claim 28, the method further comprising:

receiving a user query from a user;

determining whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

33. A system for recording, searching, and outputting display information, comprising

a display recorder that receives and stores:

display-image-state information including a first screen-image of a display at a first time and a second screen-image of the display at a second time that is after the first time;

display-command information including;

at least one first command that describes advancing the first screen-image to a third screen-image of the display at a third time that is between the first time and the second time; and

at least one second command that describes advancing the second screen-image to a fourth screen-image of the display at a fourth time that is after the third time; and

a text recorder that receives;

text information of text displayed at the third time from a text stream derived independently of the display-image-state information and independently of the display-command information; and

text information of text displayed at the fourth time from a text stream derived independently of the display-image-state information and independently of the display-command information.

34. The system of claim 33, further comprising:

a search tool that;

receives a user query; and

determines whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time; and

a display system that, based on whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time;

alters either the first screen-image by applying the at least one first command to generate a third screen-image or alters the second screen-image by applying the at least one second command to generate a fourth screen-image; and

outputs either the third screen-image or the fourth screen-image.

35. The system of claim 34,

wherein the display recorder further:

stores the first screen-image with a time stamp related to the first time;

stores the second screen-image with a time stamp related to the second time;

stores the at least one first display command with a time stamp indicating the third time; and

;

stores the at least one second display command with a time stamp indicating the fourth time; and

wherein the search tool further, based on whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time, determines to either;

alter the first screen-image by applying the at least one first command by determining the first screen-image is a screen-image recorded most closely in time to issuance of the at least one first command based on the time stamp related to the first time and the time stamp related to the second time;

or

alter the second screen-image by applying the at least one second command by determining the second screen-image is a screen-image recorded most closely in time to issuance of the at least one second command based on the time stamp related to the second time and the time stamp related to the fourth time.

time to issuance of the at least one second command based on the time stamp related to the second time and the time stamp related to the fourth time.