METHOD AND APPARATUS FOR AUTOMATED CONTENT MARKING

• US 20090055745A1
• Filed: 08/22/2007
• Published: 02/26/2009
• Est. Priority Date: 08/22/2007
• Status: Active Grant

• Current Assignee: GOTO Group Incorporated (LogMeIn, Inc.)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Christiansen, Bernd Oliver
• Granted Patent: US 8,744,238 B2
• Time in Patent Office: Days
• US Class Current: 715/723
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/782 on tape
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Methods and apparatus for automatic content marking. In one embodiment, display information is recorded, a change in the computing environment is detected, and the recorded display information is marked in response to the detected change. A human-readable description of the change associated with the mark may also be stored. A change in the computing environment may be detected by detecting an event that reflects a change in the computing environment.

Claims

1. A method for annotating recorded content, the method comprising:

(a) providing a computing device having a computing environment comprising at least one executing program;

(b) recording, at the computing device, display information in a file;

(c) detecting, at the computing device, a change in the computing environment; and

(d) marking the recorded display information in response to the detected change.

2. The method of claim 1 wherein detecting a change in the computing environment comprises detecting an event that reflects a change in the computing environment.

3. The method of claim 1 wherein the display information is an analog signal or a digital signal.

4. The method of claim 1 wherein the display information is a video signal.

5. The method of claim 1 wherein the display information comprises the output of the executing program.

6. The method of claim 1 wherein the display information comprises the output of a program executing on a second computing device.

7. The method of claim 1 wherein the display information comprises at least one set of frame buffer contents.

8. The method of claim 1 wherein the display information comprises API calls to a display subsystem.

9. The method of claim 1 wherein marking the recorded display information comprises making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

10. The method of claim 1 wherein marking the recorded display information comprises making an entry in a file separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

11. The method of claim 1 further comprising:

(e) selecting a mark from a plurality of marks associated with the recorded display information; and

(f) displaying the portion of the recorded display information associated with the selected mark.

12. The method of claim 1 further comprising:

(e) selecting marks of a specified type from a plurality of marks associated with the recorded display information; and

(f) displaying at least one portion of the recorded display information associated with at least one selected mark.

13. The method of claim 1 wherein the change in the computing environment comprises at least one of execution of a program, termination of a program, a change in the behavior of an executing program, a change in the state of an executing program, a change of foreground application, a change of application focus, a program transitioning to execution in full-screen mode, a program transitioning from execution in full-screen mode, creation of a user interface element, display of a user interface element, a change in a user interface element, destruction of a user interface element, opening a file, closing a file, a change to a file, printing a document, initiating audio playback, terminating audio playback, changing volume of audio playback, network communications, mouse input, or keyboard input.

14. The method of claim 1 wherein the change in the computing environment occurs without user action.

15. The method of claim 1 wherein the change in the computing environment reflects user action.

16. The method of claim 1 further comprising:

(e) storing a human-readable description of the change in the computing environment associated with the mark.

17. The method of claim 16 wherein the human-readable description comprises at least one of a process name, an event, a window title, a file name, or a slide number.

18. The method of claim 1 wherein (c) comprises:

(c-1) recording the state of a first part of the computing environment;

(c-2) comparing a later state of the first part of the computing environment against the recorded state; and

(c-3) detecting a difference between the recorded state and the later state.

19. The method of claim 1 further comprising displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

20. The method of claim 1 further comprising displaying information that is related to the mark in juxtaposition with at least one of the marks.

21. The method of claim 20 wherein the related information displayed is contextual advertising.

22. The method of claim 1 wherein the computing device comprises at least two visual display devices, the computing environment comprises at least two executing programs, the first executing program providing output to the first visual display device and the second executing program providing output to the second visual display device, the recorded display information comprises the output of the first executing program, and the method further comprising recording, at the computing device, the output of the second executing program to a second file.

23. An apparatus for recording and annotating content associated with a computing device having a computing environment comprising at least one executing program, the apparatus comprising:

a storage device;

a module for detecting, at the computing device, a change in the computing environment; and

a recording module for causing storage of display information in a file in the storage device, the recording module being responsive to the detection module and marking the recorded display information in response to the detected change.

24. The apparatus of claim 23 wherein the detection module detects an event that reflects a change in the computing environment.

25. The apparatus of claim 23 wherein the recording module stores display information that is an analog signal or a digital signal.

26. The apparatus of claim 23 wherein the recording module stores display information that is a video signal.

27. The apparatus of claim 23 wherein the recording module stores display information that comprises the output of the executing program.

28. The apparatus of claim 23 wherein the recording module stores display information that comprises the output of a program executing on a second computing device.

29. The apparatus of claim 23 wherein the recording module stores display information that comprises at least one set of frame buffer contents.

30. The apparatus of claim 23 wherein the recording module stores display information that comprises API calls to a display subsystem.

31. The apparatus of claim 23 wherein the recording module comprises a module for making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

32. The apparatus of claim 23 wherein the recording module comprises a module for making an entry in a file in the storage device separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

33. The apparatus of claim 23 further comprising:

a module for receiving a selection of a mark from a plurality of marks associated with the recorded display information; and

a module for displaying the portion of the recorded display information associated with the selected mark.

34. The apparatus of claim 23 further comprising:

a module for receiving a selection of a specified type of mark from a plurality of marks associated with the recorded display information; and

a module for displaying at least one portion of the recorded display information associated with at least one selected mark.

35. The apparatus of claim 23 wherein the detection module detects at least one of execution of a program, termination of a program, a change in the behavior of an executing program, a change in the state of an executing program, a change of foreground application, a change of application focus, a program transitioning to execution in full-screen mode, a program transitioning from execution in full-screen mode, creation of a user interface element, display of a user interface element, a change in a user interface element, destruction of a user interface element, opening a file, closing a file, a change to a file, printing a document, initiating audio playback, terminating audio playback, changing volume of audio playback, network communications, mouse input, or keyboard input.

36. The apparatus of claim 23 wherein the detection module detects a change in the computing environment that occurs without user action.

37. The apparatus of claim 23 wherein the detection module detects a change in the computing environment that reflects user action.

38. The apparatus of claim 23 further comprising:

a module for storing a human-readable description of the change in the computing environment associated with the mark.

39. The apparatus of claim 38 wherein the module for storing a human-readable description stores a human-readable description comprising at least one of a process name, an event, a window title, a file name, or a slide number.

40. The apparatus of claim 23 wherein the detection module comprises:

a second recording module for recording the state of a first part of the computing environment;

a comparator for comparing a later state of the first part of the computing environment against the recorded state; and

a detector detecting the difference between the recorded state and the later state.

41. The apparatus of claim 23 further comprising a module for displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

42. The apparatus of claim 23 further comprising a module for displaying information that is related to the mark in juxtaposition with at least one of the marks.

43. The method of claim 42 wherein the related information displayed is contextual advertising.

44. The apparatus of claim 23 wherein the computing device comprises at least two visual display devices, the computing environment comprises at least two executing programs, the first executing program providing output to the first visual display device and the second executing program providing output to the second visual display device, the recorded display information comprises the output of the first executing program, and the apparatus further comprises a second recording module for causing storage of the output of the second executing program in a second file in the storage device.

45. A machine-readable storage medium containing instructions for a computing device having a computing environment comprising at least one executing program, the set of instructions comprising:

instructions for recording, at the computing device, display information in a file;

instructions for detecting, at the computing device, a change in the computing environment; and

instructions for marking the recorded display information in response to the detected change.

46. The machine-readable storage medium of claim 45 wherein the instructions for detecting a change in the computing environment comprise instructions for detecting an event that reflects a change in the computing environment.

47. The machine-readable storage medium of claim 45 wherein the instructions for marking the recorded display information comprise instructions for making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

48. The machine-readable storage medium of claim 45 wherein instructions for marking the recorded display information comprise instructions for making an entry in a file separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

49. The machine-readable storage medium of claim 45 further comprising:

instructions for selecting a mark from a plurality of marks associated with the recorded display information; and

instructions for displaying the portion of the recorded display information associated with the selected mark.

50. The machine-readable storage medium of claim 45 further comprising:

instructions for selecting marks of a specified type from a plurality of marks associated with the recorded display information; and

instructions for displaying at least one portion of the recorded display information associated with at least one selected mark.

51. The machine-readable storage medium of claim 45 further comprising:

instructions for storing a human-readable description of the change in the computing environment associated with the mark.

52. The machine-readable storage medium of claim 50 wherein the human-readable description comprises at least one of a process name, an event, a window title, a file name, or a slide number.

53. The machine-readable storage medium of claim 50 wherein the instructions for detecting a change in the computing environment comprise:

instructions for recording the state of a first part of the computing environment;

instructions for comparing a later state of the first part of the computing environment against the recorded state; and

instructions for detecting a difference between the recorded state and the later state.

54. The machine-readable storage medium of claim 45, the set of instructions further comprising instructions for recording, at the computing device, display information in a second file.

55. The machine-readable storage medium of claim 45, the set of instructions further comprising instructions for displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

56. The machine-readable medium of claim 45, the set of instructions further comprising instructions for displaying information that is related to the mark in juxtaposition with at least one of the marks.

57. The machine-readable medium of claim 56, wherein the instructions for displaying information that is related to the mark in juxtaposition comprise instructions for displaying contextual advertising in juxtaposition.