Computer graphics data recording and playback system with a VCR-based graphic user interface

• US 5,748,499 A
• Filed: 09/19/1995
• Issued: 05/05/1998
• Est. Priority Date: 09/19/1995
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Trueblood, John
• Primary Examiner(s): Voeltz, Emanuel T.
• Assistant Examiner(s): Bui, Bryan
• Application Number:
  US08/530,766
• Time in Patent Office: 959 Days
• Field of Search: 364/514 A, 364/551.01, 364/DIG. 1, 364/DIG. 2, 364/280, 364/400.01, 395/326, 395/340, 395/343, 395/339, 395/618, 395/620, 395/200.76, 395/200.33, 395/701, 345/112, 345/119, 345/185, 345/189, 345/173, 345/333, 345/418, 707/104
• US Class Current: 702/187
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A method and apparatus for recording and playing back computer graphics data, including cursor movement data, and audio data which requires no X-Window command extensions. During recording, the command data stream to a display controller is intercepted, time stamped at regular intervals, and sent to and stored in a first file in a mass storage device. Cursor movement and operation data also is intercepted, time stamped and stored in a second file. During recording, display state data is generated and recorded into a third file at specified intervals. For playback, the system operator is presented with a graphic user interface resembling the control panel of a video cassette recorder. The system operator can play back the recorded information starting from a designated instant by operating the "buttons" on the control panel. For playback, the third file is read and forwarded to the display controller. The first and second files are then merged utilizing the time stamps to ensure synchronization and forwarded in a stream to the display controller for playing back the recorded events.

Claims

1. A method of recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said method comprising the steps of:

storing X-Window commands sent from said client program to said display device in a first memory file;

time stamping said X-Window commands in said first memory file with a first set of time stamps,

at fixed intervals, storing state information in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

time stamping said state information in said second memory file with a second set of time stamps;

storing in a third memory file X-Window events generated by a pointer device, said pointer device for graphically interacting with said client program and said display; and

time stamping said X-Window events in said third file with a third set of time stamps.

2. A method as set forth in claim 1 wherein said pointer device is a mouse.

3. A method as set forth in claim 1 wherein said step of time stamping said X-Window commands in said first memory file comprises the step of:

for each time interval during which at least one command was stored, storing in said first memory file, before all X-Window commands which were recorded during said interval, a time stamp identifying said predetermined time interval.

4. A method as set forth in claim 3 wherein said step of time stamping said X-Window events in said third file comprises the step of:

for each event recorded in said third memory file, storing in said third memory file, at a position preceding said event, a time stamp indicating said predetermined interval within which said event occurred.

5. A method as set forth in claim 4 wherein said method further comprises playing back a recorded X-Window session, said method further comprising the steps of:

(a) determining a start time for said playback;

(b) determining the first time stamp in said event file subsequent to said start time;

(c) determining the first time stamp in said X-command file subsequent to said start time;

(d) synchronizing said time stamps to a clock of a host processor;

(e) whenever said clock reaches a time corresponding to a determined time stamp in said third memory file, reading out said event which corresponds to said time stamp to said display device;

(f) determining the time stamp in said event file of a next event in said file;

(g) when said clock reaches a time corresponding to a determined time stamp in said X-command file, reading out all X-commands corresponding to said time stamp to said display device;

(h) determining a next time stamp in said X-command file; and

(i) repeating steps (e)-(h).

6. A method as set forth in claim 1 wherein said method further comprises playing back a recorded X-Windows session, said method further comprising the steps of:

reading out to said display device said X-Window commands stored in said first memory file using said first set of time stamps to read out said commands with a relative timing corresponding to a relative timing at which they were stored; and

reading out to said display device said X-Window events stored in said second memory file using said time stamps to synchronize to said reading out of X-Window commands from said first memory file.

7. A method as set forth in claim 6 further comprising the steps of providing a graphic user interface in the form of a video cassette recorder face plate as a display on said display device for initiating said playing back of a recording session and setting playback parameters.

8. A method of recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said method comprising the steps of:

storing X-Window commands sent from said client program to said display device in a first memory file;

time stamping said X-Window commands in said first memory file with a first set of time stamps, at fixed intervals, storing state information in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

time stamping said state information in said second memory file with a second set of time stamps;

wherein said step of storing state information comprises storing X-Window CREATE commands; and

further

wherein said step of storing state information further comprises;

intercepting each X-Window command and event directed to said display device;

determining which, if any, of said stored CREATE commands have parameters which are affected by said intercepted command or event; and

modifying said parameters of said stored CREATE commands which are affected by said intercepted command or event in accordance with said intercepted command or event.

9. A method of recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said method comprising the steps of:

storing X-Window commands sent from said client program to said display device in a first memory file;

time stamping said X-Window commands in said first memory file with a first set of time stamps, at fixed intervals, storing state information in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

time stamping said state information in said second memory file with a second set of time stamps;

wherein said work station further comprises audio means for providing audio data corresponding to said display on said display device, said method further comprising the steps of;

storing in a fourth memory file said audio data; and

storing in a fifth memory file time stamps corresponding to said audio data in said fourth memory file.

10. The method of claim 9

wherein said step of storing X-Window commands comprises the steps of intercepting X-Window commands sent to said display device and forwarding said commands to said first memory file and to said display device.

11. A method of recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said method comprising the steps of:

storing X-Window commands sent from said client program to said display device in a first memory file;

time stamping said X-Window commands in said first memory file with a first set of time stamps, at fixed intervals, storing state information in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

time stamping said state information in said second memory file with a second set of time stamps;

wherein said step of storing state information comprises the steps of;

identifying commands which may affect the state of the display;

identifying parameters of the identified commands which may alter the current state of the display; and

updating a table of X-Window CREATE commands with state-affecting parameters in accordance with said identified parameters.

12. A method as set forth in claim 11 further comprising the steps of:

immediately after said step of storing said state information, issuing EXPOSE X-Window events for each window which is active in said display.

13. A method as set forth in claim 12 wherein said method further comprises playing back a recorded X-Windows session from a random starting time, said method further comprising the steps of:

reading out to said display device said create commands;

reading out to said display device said X-Window commands stored in said first memory file starting at said random starting time using said first set of time stamps to read out said commands with a relative timing corresponding to a relative timing at which they were stored; and

reading out to said display device said X-Window events stored in said second memory file using said time stamps to synchronize to said reading out of X-Window commands from said first memory file.

14. A method as set forth in claim 13 further comprising the steps of:

if said specified starting time is other than at the beginning of a recording session, prior to step (e), reading out said state information from said second memory file corresponding to an instant preceding said starting time by a predetermined interval; and

reading out to said display device said X-Window commands stored in said first memory file in the order in which they were stored from said instant to said specified starting time.

15. An apparatus for recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said apparatus comprising:

a work station including a graphic display device for generating a display;

means for intercepting X-Window commands sent from said client program to said display device;

means for storing said X-Window commands in a first memory file with a first set of periodic time stamps;

means for storing state information at fixed intervals in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

a pointer device for graphically interacting with said client program and said display;

means for storing in a third memory file X-Window events generated by said pointer device; and

means for time stamping said X-Window events in said third file with a third set of time stamps.

16. A method as set forth in claim 15 wherein said pointer device is a mouse.

17. An apparatus as set forth in claim 15 wherein said means for storing said X-Window commands in said first memory file comprises means for storing in said first memory file, each time interval of a predetermined duration during which interval at least one X-Window command was issued, a time stamp indicating said predetermined time interval followed by all X-Window commands which were recorded during said interval.

18. An apparatus as set forth in claim 17 wherein said means for storing said X-Window events in said third file comprises means for storing in said third memory file, for each event recorded in said third memory file, a time stamp indicating said predetermined interval within which said event occurred followed by said event.

19. An apparatus as set forth in claim 18 further comprising means for playing back a recorded X-Window session, said method further comprising:

(a) means for determining a start time for said playback;

(b) means for determining the first time stamp in said event file subsequent to said start time;

(c) means for determining the first time stamp in said X-command file subsequent to said start time;

(d) means for synchronizing said time stamps to a clock of a host processor;

(e) means for reading out said event which corresponds to said time stamp and executing said event, when said clock reaches a time corresponding to a determined time stamp in said event file;

(f) means for determining the time stamp in said event file of a next event in said file;

(g) means for reading out all X-commands corresponding to said time stamp and executing said commands, when said clock reaches a time corresponding to a determined time stamp in said X-command file;

(h) means for determining a next time stamp in said X-command file; and

(i) means for repeating steps (e)-(h).

20. An apparatus for recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said apparatus comprising:

a work station including a graphic display device for generating a display;

means for intercepting X-Window commands sent from said client program to said display device;

means for storing said X-Window commands in a first memory file with a first set of periodic time stamps; and

means for storing state information at fixed intervals in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

wherein said means for storing state information comprises;

means for maintaining a set of state generating commands for generating windows in said display;

means for determining those X-Window commands which have an effect on the state of the display; and

means for updating said state generating commands in accordance with said determined X-Window commands.

21. An apparatus as set forth in claim 20 further comprising a mass storage device in which said memory files are stored.

22. An apparatus as set forth in claim 21 wherein said mass storage device comprises a hard disk.

23. An apparatus as set forth in claim 22 further comprises means for playing back a recorded X-Windows session, said means for playing back comprising:

means for reading out to said display device said X-Window commands stored in said first memory file starting at a specified starting time using said first set of time stamps to read out said command with a relative timing corresponding to a relative timing at which they were stored;

means for reading out to said display device said X-Window events stored in said second memory file using said time stamps to synchronize to said reading out of X-Window commands from said first memory file; and

means for reading out to an audio reproduction device said audio information stored in said fourth memory file using said time stamps stored in said fifth memory file to synchronize said audio data to said reading out of X-Window commands from said first memory file.

24. An apparatus as set forth in claim 23 further comprising:

means for determining if said specified start time is other than at the beginning of a recorded session;

means for reading out to said display device said state information from said second memory file corresponding to an instant preceding said starting time by a predetermined interval prior to reading out said data from said first memory file, if said specified starting time is other than at the beginning of a recording session; and

means for reading out to said display device said X-Window commands stored in said first memory file in the order in which they were stored from said instant to said specified starting time.

25. An apparatus as set forth in claim 24 wherein said predetermined interval is at least about 2 seconds.

26. An apparatus as set forth in claim 20 wherein said work station further comprises:

audio means for providing audio data corresponding to said display;

means for storing in a fourth memory file said audio data; and

means for storing in a fifth memory file time stamps corresponding to said audio data in said fourth memory file.

27. An apparatus for recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said apparatus comprising:

a work station including a graphic display device for generating a display;

means for intercepting X-Window commands sent from said client program to said display device;

means for storing said X-Window commands in a first memory file with a first set of periodic time stamps; and

means for storing state information at fixed intervals in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

wherein said means for storing state information comprises;

means for identifying commands which may affect the state of the display;

means for identifying parameters of the identified commands which may alter the current state of the display; and

means for updating a table of state-affecting parameters in accordance with said identified parameters.

28. An apparatus as set forth in claim 27 further comprising means for issuing EXPOSE X-Window events for each window which is active in said display immediately after said step of storing said state information, so as to re-draw said windows.

29. An apparatus for recording an X-Window session in which X-Window commands and events create a display on a display device that is coupled to a client program through a work station, said apparatus comprising:

a work station including a graphic display device for generating a display;

means for intercepting X-Window commands sent from said client program to said display device;

means for storing said X-Window commands in a first memory file with a first set of periodic time stamps;

means for storing state information at fixed intervals in a second memory file, said state information comprising information from which the state of the display at said fixed interval can be determined;

means for playing back a recorded X-Windows session, said means for playing back comprising;

means for reading out to said display device said X-Window commands stored in said first memory file starting at a specified starting time using said first set of time stamps to read out said commands with a relative timing corresponding to a relative timing at which they were stored; and

means for reading out to said display device said X-Window events stored in said second memory file using said time stamps to synchronize to said reading out of X-Window commands from said first memory file.

30. An apparatus as set forth in claim 29 wherein said display device comprises:

an X-server coupled to receive X-Window commands and events from said means for intercepting and from said means for playing back;

a graphic controller card coupled to receive data from said X-server; and

a display terminal.

31. An apparatus as set forth in claim 29 further comprising:

means for providing a graphic user interface on said display device in the form of a video cassette recorder for initiating said playing back of a recording session and setting playback parameters.