Method and apparatus for automated content marking

• US 8,744,238 B2
• Filed: 08/22/2007
• Issued: 06/03/2014
• Est. Priority Date: 08/22/2007
• Status: Active Grant

• Current Assignee: GOTO Group Incorporated (LogMeIn, Inc.)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Christiansen, Bernd Oliver
• Primary Examiner(s): NGUYEN, HUY THANH
• Application Number:
  US11/843,345
• Publication Number:
  US 20090055745A1
• Time in Patent Office: 2,477 Days
• US Class Current: 386/241
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/782 on tape
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Methods and apparatus for automatic content marking. In one embodiment, display information is recorded, a change in the computing environment is detected, and the recorded display information is marked in response to the detected change. A human-readable description of the change associated with the mark may also be stored. A change in the computing environment may be detected by detecting an event that reflects a change in the computing environment.

Claims

1. A method for textually annotating recorded content, the method comprising:

at a computing device having a computing environment comprising at least one executing program, recording display information, relating to an output of the at least one executing program, in a file;

detecting a change of a display state in the computing environment effecting the display information by;

receiving a notification in a case where the change of the display state triggers a notification to be provided, and

querying a current display state and comparing the current display state to a stored record of a previous display state in a case where the change of the display state does not trigger the notification to be provided; and

textually annotating the recorded display information, with information indicating the detected change, in response to the detected change.

2. The method of claim 1 wherein detecting a change in the computing environment comprises detecting an event that reflects a change in the computing environment.

3. The method of claim 1 wherein the display information is an analog signal or a digital signal.

4. The method of claim 1 wherein the display information is a video signal.

5. The method of claim 1 wherein the display information comprises the output of the executing program.

6. The method of claim 1 wherein the display information comprises the output of a program executing on a second computing device.

7. The method of claim 1 wherein the display information comprises at least one set of frame buffer contents.

8. The method of claim 1 wherein the display information comprises API calls to a display subsystem.

9. The method of claim 1 wherein textually annotating the recorded display information comprises making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

10. The method of claim 1 wherein textually annotating the recorded display information comprises making an entry in a file separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

11. The method of claim 1 further comprising:

selecting a mark from a plurality of marks associated with the recorded display information; and

displaying the portion of the recorded display information associated with the selected mark.

12. The method of claim 1 further comprising:

selecting marks of a specified type from a plurality of marks associated with the recorded display information; and

displaying at least one portion of the recorded display information associated with at least one selected mark.

13. The method of claim 1 wherein the change in the computing environment comprises at least one of execution of a program, termination of a program, a change in the behavior of an executing program, a change in the state of an executing program, a change of foreground application, a change of application focus, a program transitioning to execution in full-screen mode, a program transitioning from execution in full-screen mode, creation of a user interface element, display of a user interface element, a change in a user interface element, destruction of a user interface element, opening a file, closing a file, a change to a file, printing a document, initiating audio playback, terminating audio playback, changing volume of audio playback, network communications, mouse input, or keyboard input.

14. The method of claim 1 wherein the change in the computing environment occurs without user action.

15. The method of claim 1 wherein the change in the computing environment reflects user action.

16. The method of claim 1 further comprising:

storing a human-readable description of the change in the computing environment associated with the detected change.

17. The method of claim 16 wherein the human-readable description comprises at least one of a process name, an event, a window title, a file name, or a slide number.

18. The method of claim 1 further including detecting the change in the computing environment by:

recording an initial state of a part of the computing environment;

comparing a later state of the part of the computing environment against the initial state; and

detecting a difference between the initial state and the later state.

19. The method of claim 1 further comprising selecting marks of a specified type from a plurality of marks associated with the recorded display information and displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

20. The method of claim 1 further comprising selecting marks of a specified type from a plurality of marks associated with the recorded display information and displaying information that is related to the mark in juxtaposition with at least one of the marks.

21. The method of claim 20 wherein the related information displayed is contextual advertising.

22. The method of claim 1 wherein the computing device comprises at least two visual display devices, the computing environment comprises at least two executing programs, the first executing program providing an output to the first visual display device and the second executing program providing an output to the second visual display device, the recorded display information comprises the output of the first executing program, and the method further comprising recording, at the computing device, the output of the second executing program to a second file.

23. An apparatus for recording and textually annotating content associated with a computing device having a computing environment comprising at least one executing program, the apparatus comprising:

a storage device;

a detection module, at the computing device, for detecting a change of a display state in the computing environment effecting display information, relating to an output of the at least one executing program, the detection module operating by;

receiving a notification in a case where the change of the display state triggers a notification to be provided, and

querying a current display state and comparing the current display state to a stored record of a previous display state in a case where the change of the display state does not trigger the notification to be provided; and

a recording module for causing storage of the display information in a file in the storage device, the recording module being responsive to the detection module and textually annotating the recorded display information, with information indicating the detected change, in response to the detected change.

24. The apparatus of claim 23 wherein the detection module detects an event that reflects a change in the computing environment.

25. The apparatus of claim 23 wherein the recording module stores display information that is an analog signal or a digital signal.

26. The apparatus of claim 23 wherein the recording module stores display information that is a video signal.

27. The apparatus of claim 23 wherein the recording module stores display information that comprises the output of the executing program.

28. The apparatus of claim 23 wherein the recording module stores display information that comprises the output of a program executing on a second computing device.

29. The apparatus of claim 23 wherein the recording module stores display information that comprises at least one set of frame buffer contents.

30. The apparatus of claim 23 wherein the recording module stores display information that comprises API calls to a display subsystem.

31. The apparatus of claim 23 wherein the recording module comprises a module for making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

32. The apparatus of claim 23 wherein the recording module comprises a module for making an entry in a file in the storage device separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

33. The apparatus of claim 23 further comprising:

a module for receiving a selection of a mark from a plurality of marks associated with the recorded display information; and

a module for displaying the portion of the recorded display information associated with the selected mark.

34. The apparatus of claim 23 further comprising:

a module for receiving a selection of a specified type of mark from a plurality of marks associated with the recorded display information; and

a module for displaying at least one portion of the recorded display information associated with at least one selected mark.

35. The apparatus of claim 23 wherein the detection module detects at least one of execution of a program, termination of a program, a change in the behavior of an executing program, a change in the state of an executing program, a change of foreground application, a change of application focus, a program transitioning to execution in full-screen mode, a program transitioning from execution in full-screen mode, creation of a user interface element, display of a user interface element, a change in a user interface element, destruction of a user interface element, opening a file, closing a file, a change to a file, printing a document, initiating audio playback, terminating audio playback, changing volume of audio playback, network communications, mouse input, or keyboard input.

36. The apparatus of claim 23 wherein the detection module detects a change in the computing environment that occurs without user action.

37. The apparatus of claim 23 wherein the detection module detects a change in the computing environment that reflects user action.

38. The apparatus of claim 23 further comprising:

a module for storing a human-readable description of the change in the computing environment associated with the detected change.

39. The apparatus of claim 38 wherein the module for storing a human-readable description stores a human-readable description comprising at least one of a process name, an event, a window title, a file name, or a slide number.

40. The apparatus of claim 23 wherein the detection module comprises:

a second recording module for recording an initial state of a part of the computing environment;

a comparator for comparing a later state of the part of the computing environment against the initial state; and

a detector detecting the difference between the initial state and the later state.

41. The apparatus of claim 23 further comprising:

a module for receiving a selection of a specified type of mark from a plurality of marks associated with the recorded display information; and

a module for displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

42. The apparatus of claim 23 further comprising:

a module for receiving a selection of a specified type of mark from a plurality of marks associated with the recorded display information; and

a module for displaying information that is related to the mark in juxtaposition with at least one of the marks.

43. The method of claim 42 wherein the related information displayed is contextual advertising.

44. The apparatus of claim 23 wherein the computing device comprises at least two visual display devices, the computing environment comprises at least two executing programs, the first executing program providing an output to the first visual display device and the second executing program providing an output to the second visual display device, the recorded display information comprises the output of the first executing program, and the apparatus further comprises a second recording module for causing storage of the output of the second executing program in a second file in the storage device.

45. A non-transitory machine-readable storage medium containing instructions for a computing device having a computing environment comprising at least one executing program, the set of instructions comprising:

instructions for recording, at the computing device, display information, relating to an output of the at least one executing program, in a file;

instructions for detecting, at the computing device, a change of a display state in the computing environment effecting the display information by;

receiving a notification in a case where the change of the display state triggers a notification to be provided, and

querying a current display state and comparing the current display state to a stored record of a previous display state in a case where the change of the display state does not trigger the notification to be provided; and

instructions for textually annotating the recorded display information, with information indicating the detected change, in response to the detected change.

46. The non-transitory machine-readable storage medium of claim 45 wherein the instructions for detecting a change in the computing environment comprise instructions for detecting an event that reflects a change in the computing environment.

47. The non-transitory machine-readable storage medium of claim 45 wherein the instructions for marking the recorded display information comprise instructions for making an entry in the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

48. The non-transitory machine-readable storage medium of claim 45 wherein instructions for marking the recorded display information comprise instructions for making an entry in a file separate from the file comprising the recorded display information such that a portion of the recorded display information substantially contemporaneous with the change in the computing environment is associated with the entry.

49. The non-transitory machine-readable storage medium of claim 45 further comprising:

instructions for selecting a mark from a plurality of marks associated with the recorded display information; and

instructions for displaying the portion of the recorded display information associated with the selected mark.

50. The non-transitory machine-readable storage medium of claim 45 further comprising:

instructions for selecting marks of a specified type from a plurality of marks associated with the recorded display information; and

instructions for displaying at least one portion of the recorded display information associated with at least one selected mark.

51. The non-transitory machine-readable storage medium of claim 45 further comprising:

instructions for storing a human-readable description of the change in the computing environment associated with the detected change.

52. The non-transitory machine-readable storage medium of claim 50 wherein the human-readable description comprises at least one of a process name, an event, a window title, a file name, or a slide number.

53. The non-transitory machine-readable storage medium of claim 50 wherein the instructions for detecting a change in the computing environment comprise:

instructions for recording an initial state of a part of the computing environment;

instructions for comparing a later state of the part of the computing environment against the initial state; and

instructions for detecting a difference between the initial state and the later state.

54. The non-transitory machine-readable storage medium of claim 45, the set of instructions further comprising instructions for recording, at the computing device, display information in a second file.

55. The non-transitory machine-readable storage medium of claim 45, the set of instructions further comprising instructions for selecting a mark from a plurality of marks associated with the recorded display information and instructions for displaying at least part of the recorded display information in juxtaposition with at least one of the marks.

56. The non-transitory machine-readable medium of claim 45, the set of instructions further comprising instructions for selecting a mark from a plurality of marks associated with the recorded display information and instructions for displaying information that is related to the mark in juxtaposition with at least one of the marks.

57. The non-transitory machine-readable medium of claim 56, wherein the instructions for displaying information that is related to the mark in juxtaposition comprise instructions for displaying contextual advertising in juxtaposition.