Systems, methods, means, and media for recording, searching, and outputting display information

• US 8,214,367 B2
• Filed: 02/27/2008
• Issued: 07/03/2012
• Est. Priority Date: 02/27/2007
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Laadan, Oren, Phung, Dan, Nieh, Jason, Baratto, Ricardo, Potter, Shaya Joseph
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US12/038,515
• Publication Number:
  US 20080301101A1
• Time in Patent Office: 1,588 Days
• Field of Search: 1/1, 707/812, 707/706, 707/736, 707/758, 707/648, 707/739, 707/999.001, 702/187, 714/37, 715/733, 715/736, 715/764
• US Class Current: 707/739
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Systems for recording, searching, and outputting display information are provided. In some embodiments, systems for recording display information are provided. The systems include a virtual display that: intercepts display-changes describing changes to be made to a state of a display; sends the display-changes to a client for display; records the display-changes; and a context recorder that records context information describing a state of the display derived from a source independently of the display changes and independently of screen-images. In some embodiments, the systems further include a display system that generates an output screen-image based at least in part on at least one of the display-changes and in response to a search of the context information. In some embodiments, the virtual display further records screen-images; and the display system further generates the output screen-image based at least in part on a recorded-screen-image of the recorded screen-images.

Claims

1. A system for recording display information, comprising:

a virtual display that;

intercepts display-changes describing changes to be made to a state of a display;

sends the display-changes to a client for display; and

records the display-changes;

a context recorder that uses accessibility mechanisms to record context information derived independently of screen-images,

wherein the accessibility mechanisms include mechanisms utilized by a screen reader for providing desktop access for visually impaired individuals; and

wherein the context information includes (1) text that is displayed in at least one application window, (2) a duration in which the text is displayed on the at least one application window and (3) at least one of a font type of the text, a font size of the text, and an identifier of the at least one application window, that are extracted through the accessibility mechanisms;

a search tool that;

receives a user query from a user;

determines whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, determines a correspondence between context information that matched the query and at least one of the display changes; and

a display system that;

generates an output screen-image based at least in part on the at least one of the display-changes determined to have a correspondence with the context information that matched the query.

2. The system of claim 1, wherein the display-changes are display commands.

3. The system of claim 1, further comprising:

an execution recorder that records execution of an application generating the display; and

an execution revival system that revives execution from a time of the output screen-image.

4. A method for recording display information, comprising:

intercepting display-changes describing changes to be made to a state of a display;

sending the display-changes to a client for display;

recording the display-changes;

recording context information using accessibility mechanisms,

wherein the context information is derived independently of screen-images;

wherein the accessibility mechanisms include mechanisms utilized by a screen reader for providing desktop access for visually impaired individuals; and

wherein the context information includes (1) text that is displayed in at least one application window, (2) a duration in which the text is displayed on the at least one application window and (3) at least one of a font type of the text, a font size of the text, and an identifier of the at least one application window, that are extracted through the accessibility mechanisms;

receiving a user query from a user;

determining whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

5. The method of claim 4, wherein the display-changes are display commands.

10. The method of claim 4, further comprising:

recording execution of an application generating the display; and

reviving execution from a time of the output screen-image.

6. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for recording display information, the method comprising:

intercepting display-changes describing changes to be made to a state of a display;

sending the display-changes to a client for display;

recording the display-changes;

recording context information using accessibility mechanisms,

wherein the context information is derived independently of screen-images;

wherein the accessibility mechanisms include mechanisms utilized by a screen reader for providing desktop access for visually impaired individuals; and

wherein the context information includes (1) text that is displayed in at least one application window, (2) a duration in which the text is displayed on the at least one application window and (3) at least one of a font type of the text, a font size of the text, and an identifier of the at least one application window, that are extracted through the accessibility mechanisms;

receiving a user query from a user;

determining whether the user query matches any of the context information; and

based on the whether the user query matches any of the context information, outputting at least one result describing a state of the display generated based at least in part on at least one of the display-changes.

7. The computer-readable medium of claim 6, wherein the display-changes are display commands.

11. The computer-readable medium of claim 6, wherein the method further comprising:

recording execution of an application generating the display; and

reviving execution from a time of the output screen-image.

8. A system for recording, searching, and outputting display information, comprising:

a display recorder that receives and stores;

display-image-state information including a first screen-image of a display at a first time and a second screen-image of the display at a second time that is after the first time;

display-command information including;

at least one first command that describes advancing the first screen-image to a third screen-image of the display at a third time that is between the first time and the second time; and

at least one second command that describes advancing the second screen-image to a fourth screen-image of the display at a fourth time that is after the third time;

a text recorder that receives;

text displayed in at least one application window at the third time, wherein the text displayed at the third time is extracted independently of screen-images through accessibility mechanisms and wherein the accessibility mechanisms include mechanisms utilized by a screen reader for providing desktop access for visually impaired individuals; and

text displayed in at least one application window at the fourth time, wherein the text displayed at the fourth time is extracted independently of screen-images through the accessibility mechanisms;

a search tool that;

receives a user query; and

determines whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time; and

a display system that, based on whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time;

alters either the first screen-image by applying the at least one first command to generate a third screen-image or alters the second screen-image by applying the at least one second command to generate a fourth screen-image; and

outputs either the third screen-image or the fourth screen-image.

9. The system of claim 8,

wherein the display recorder further:

stores the first screen-image with a time stamp related to the first time;

stores the second screen-image with a time stamp related to the second time;

stores the at least one first display command with a time stamp indicating the third time; and

stores the at least one second display command with a time stamp indicating the fourth time; and

wherein the search tool further, based on whether the user query matches the text displayed at the third time or matches the text displayed at the fourth time, determines to either;

alter the first screen-image by applying the at least one first command by determining the first screen-image is a screen-image recorded most closely in time to issuance of the at least one first command based on the time stamp related to the first time and the time stamp related to the second time;

or

alter the second screen-image by applying the at least one second command by determining the second screen-image is a screen-image recorded most closely in time to issuance of the at least one second command based on the time stamp related to the second time and the time stamp related to the fourth time.