Method, system, and computer-readable recording medium for synchronous multi-media recording and playback with end user control of time, data, and event visualization for playback control over a network

• US 8,113,844 B2
• Filed: 12/15/2006
• Issued: 02/14/2012
• Est. Priority Date: 12/15/2006
• Status: Active Grant

• Current Assignee: Atellis, Inc.
• Original Assignee: Atellis, Inc.
• Inventors: Huang, Lucas K., Kazoun, Chafic A.
• Primary Examiner(s): Hu, Kang
• Application Number:
  US11/611,792
• Publication Number:
  US 20080145830A1
• Time in Patent Office: 1,887 Days
• Field of Search: 434/246, 434/262, 434/322, 434/365, 707/809
• US Class Current: 434/262
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A system and system are provided for providing synchronous multimedia recording and playback. Aspects the exemplary embodiment include in response to a training session is being conducted, synchronously recording in real-time simulator data from a simulator captured by a simulator capture tool, and video of the training session captured by a plurality of A/V sources; encoding each of the videos captured by the plurality of A/V sources as respective digital media files formatted as streaming media; and transmitting both the simulator data and the video media files from a server to a client over a network, such that when the client receives the simulator data and the stream, the respective videos are synchronously played back with the simulator data on the client.

Claims

1. A method for providing synchronous multimedia recording and playback, comprising:

synchronously recording in real-time, simulator data from a simulator captured by a simulator capture tool, and video of session captured by at least one audio/video source;

encoding the recorded video captured as a digital media file wherein the digital media file is formatted as streaming media;

transmitting the simulator data and the streaming media from a server to a client over a network;

integrating the streaming media and the simulator data for playback in a single web browser; and

synchronously playing the streaming media with the simulator data in the single web browser on the client,

wherein the synchronously playing comprises linking the streaming media with the simulator data such that the streaming media and the simulator data are both displayed with reference to a timeline and such that when moving to a particular point on the timeline, corresponding streaming media and simulator data are played from said particular point on the timeline.

2. The method of claim 1, wherein said synchronously recording comprises recording in real-time the video of the session being captured by a plurality of audio/video sources that include the at least one audio/video source and wherein said streaming comprises playing back in real time respective videos from the plurality of audio/video sources.

3. The method of claim 2, wherein the single web browser is divided into respective areas for each video.

4. The method of claim 3, wherein the streaming further comprises dynamically generating a composite view that references as sources respective streaming media files obtained by recording in real-time the video of a training session being captured by the plurality of audio/video sources, and streaming a single media file.

5. The method of claim 2, further comprising designating a primary audio source selected from one of the respective videos, turning off audio portion of remaining respective videos, and synchronizing, based on time, video with the primary audio source with the remaining respective videos and the simulator data.

6. The method of claim 2, wherein:

the respective videos comprise at least one video feed and at least one audio feed,

input from a user comprises at least one of text annotations, graphical annotations, assessments, posted questions, and comments;

the respective videos are seamlessly assembled in real-time and are streamed to at least one user, wherein the respective videos are streamed or obtained using at least one of internet, encoding device, a centralizing server, or locally or remotely stored video; and

streaming media obtained from the respective videos and the simulator data are viewable in real-time and after said synchronous recording using the web browser with a multimedia player.

7. The method of claim 2, further comprising storing the simulator data and digital media files obtained from the respective videos, and transmitting the synchronous playback on-demand, subsequent to the training session.

8. The method of claim 7, further comprising prior to the synchronous playback, allowing a user of the client to select which ones of the respective videos are played back.

9. The method of claim 2, further comprising enabling a user to enter questions, comments, annotations, and assessment data while the respective videos and the simulator data are synchronously played back.

10. The method of claim 9, further comprising during the synchronous playback in real-time, displaying a checklist of training tasks to be completed by a user, that indicate which tasks were completed during the synchronous playback for real-time annotation and assessment of a trainee.

11. The method of claim 10, wherein the stream is received and played back on multiple web clients to provide multiple simultaneous local and remote assessments by multiple users.

12. The method of claim 1, further comprising recording the simulator data as captured video of the simulator showing an output of the simulator.

13. The method of claim 12, wherein the video of the simulator data is also encoded as a streaming media file, and the video of the simulator data and the video of the AN source are played back together in the single web browser.

14. The method of claim 1, further comprising recording the simulator data as telemetry values.

15. The method of claim 14, further comprising playing back the video and the simulator data in a multimedia player of the client.

16. The method of claim 14, further comprising playing back the simulator data as a graph line of the telemetry values along the timeline.

17. The method of claim 16, further comprising identifying transition points of the telemetry values during playback.

18. The method of claim 16, further comprising allowing a user to selectively control visualization of the telemetry information by selecting which data points to display during playback.

19. The method of claim 16, further comprising displaying points of interest along the timeline.

20. The method of claim 1, further comprising allowing a user to advance or return to any point in time in the synchronous play back of the video while playing by interacting with the timeline.

21. The method of claim 1, further comprising streaming the synchronous playback in real-time during the session such that the simulator data and the streaming media are played back synchronously in real-time.

50. The method of claim 1, wherein the timeline further links annotations with the stimulator data and the streaming media.

51. The method of claim 1, further comprising a time sync generator that is coupled to the simulator and the simulator capture tool and that controls synchronization of the recordings.

52. The method of claim 1, wherein the time sync generator calculates relative time of the simulator data with respect to the streaming media.

53. The method of claim 1, wherein all of participants that interact with the simulator in the session are present in the room.

54. The method of claim 1, further comprising mixing the various video data prior to the synchronous playback forming streamed composite media file.

55. The method of claim 54, further comprising dynamically mixing the various video data prior to the synchronous playback forming streamed composite media file.

56. The method of claim 54, wherein the simulator data and the streamed composite media file are synchronously played back based on time of the recordings.

57. The method of claim 10, wherein the checklist is displayed in an area within the single web browser and the real-time annotations are applied to the streaming media and the simulator data being synchronously played-back in the single web browser.

22. A system for providing synchronous multimedia recording and playback, comprising:

at least one audio/video source which captures video of a session, the session including a use of a simulator;

a simulation capture tool which captures real-time simulator data from the simulator, wherein the simulator data and the video of the session are recorded synchronously;

one or more encoders which encode the video captured by the at least one audio/video source as a digital media file, wherein the digital media file is formatted as streaming media;

a transmitter which transmits the simulator data and the streaming media from a server to a client over a network;

an integration module which integrates the streaming media with the simulator data for playback in a single web browser; and

a playback module which plays back the received streaming media file in synchronization with the simulator data in the single web browser on the client,

wherein the integration module links the streaming media with the simulator data such that the streaming media and the simulator data are both displayed with reference to a timeline by the playback module and such that when moving to a particular point on the timeline, corresponding streaming media and simulator data are played by the playback module from said particular point on the timeline.

23. The system of claim 22, wherein the playback module comprises a assessment tool, and

wherein the assessment tool comprises:

an annotation and assessment tool which enables a user to enter questions, comments, annotation and assessment data of performance of a participant while the video and simulation data are synchronously played back by the playback module; and

a debrief tool which enables automated debrief session by allowing the user to select which recorded training session to view.

24. The system of claim 22, further comprising a plurality of audio/video sources including the at least one audio/video source, wherein respective videos from the plurality of audio/video sources are played back by the playback module in real time.

25. The system of claim 24, wherein the single browser window is divided into respective areas for each video.

26. The system of claim 22, wherein the playback module allows a user to advance or return to any point in time in the synchronous play back of the videos while playing by interacting with the timeline.

27. The system of claim 24, wherein prior to the synchronous playback by the playback module, a selection module allows a user to select which ones of the videos are played back.

28. The system of claim 24, wherein the simulator data is recorded by the simulation capture tool as captured video of the simulator showing an output of the simulator.

29. The system of claim 28, wherein the video of the simulator data is also encoded by the one or more encoders as a streaming media file, and the video of the simulator data and the respective videos are played back in the single web browser by the playback module.

30. The system of claim 24, wherein the simulator data is recorded by the simulation capture tool as telemetry values.

31. The system of claim 30, wherein the simulator data is played back by the playback module as a graph line of the telemetry values along the timeline.

32. The system of claim 31, wherein transition points of the telemetry values are identified during playback by the playback module.

33. The system of claim 31, further comprising a user interface that receives user input in which the user selectively controls visualization of the telemetry information.

34. The system of claim 31, wherein points of interest are displayed along the timeline during playback by the playback module.

35. The system of claim 30, wherein the video is played back and the simulator data is played back by the playback module in a multimedia player of the client.

36. The system of claim 22, wherein the synchronous playback is streaming in real-time during the session by the one or more encoders.

37. The system of claim 22, further comprising a storage module which stores the simulator data as streaming media files, wherein the one or more encoders stream the synchronous playback subsequent to the session.

38. The system of claim 22, wherein during the synchronous playback in real-time by the playback module, the playback module displays a checklist of training tasks to be completed by a user is displayed, and further comprising an input module which allows a user to indicate which tasks were completed during the synchronous playback for real-time annotation and assessment of the trainee.

39. The system of claim 22, wherein the stream is received and played back by the playback module on multiple web clients to provide multiple simultaneous local and remote assessments by multiple users.

40. The system of claim 22, wherein the playback module further comprises an assessment administration interface that allows input for pre, live, and post assessment of the session.

41. The system of claim 22, wherein the playback module synchronizes and visually links the timeline with other data comprising captured computer screens or charted data from the simulator allowing the user to navigate the timeline.

42. The system of claim 22, further comprising an annotation module that allows the user to post questions or comments to other users that are visually represented on the timeline.

43. The system of claim 22, further comprising an interface module which generates an interface in which the respective user documents sessions, annotations, posts, and usage statistics.

44. The system of claim 22, wherein the at least one encoder indexes the video such that text in the video is searchable.

45. The system of claim 22, the playback module further comprises an interface module which allows a user to associate individualized assessments to participant types.

46. The system of claim 22, wherein the simulation capture tool is at least one audio/video source that captures the simulator data.

47. An executable software product stored on a non-transitory computer-readable medium containing program instructions for providing synchronous multimedia recording and playback, wherein video of a session is captured by at least one audio/video source, the program instructions comprising:

synchronously recording with the video, real-time simulator data from a simulator captured by a simulator capture tool;

encoding the video captured by the at least one audio/video source as a digital media file, wherein the digital medial file is formatted as streaming media;

transmitting the simulator data and the video media file from a server to a client over a network;

integrating the streaming media and the simulator data for playback in a single web browser; and

playing back the video in synchronization with the simulator data in the single web browser,

wherein the playing back comprises linking the streaming media with the simulator data such that the streaming media and the simulator data are both displayed with reference to a timeline and such that when moving to a particular point on the timeline, corresponding streaming media and simulator data are played from said particular point on the timeline.

48. A method for providing synchronous multimedia recording and playback, comprising:

synchronously recording in real-time data output from a medical device, and video of a medical procedure related to the medical device, wherein the medical procedure is captured by at least one audio/video source;

encoding the recorded video captured as a digital media file wherein the digital media file is formatted as streaming media;

transmitting the data and the streaming media from a server to a client over a network;

integrating the streaming media and the simulator data for playback in a single web browser; and

synchronously playing the streaming media with the data in the single web browser on the client,

wherein the synchronously playing comprises linking the streaming media with the simulator data such that the streaming media and the simulator data are both displayed with reference to a timeline and such that when moving to a particular point on the timeline, corresponding streaming media and simulator data are played from said particular point on the timeline.

49. A system for providing synchronous multimedia recording and playback, the system comprising:

means for synchronously recording in real-time data output from a simulator or medical device and video of a corresponding procedure related to the medical device or the simulator, wherein the corresponding procedure is captured by at least one audio/video source;

means for encoding the recorded video captured as a digital media file wherein the digital media file is formatted as streaming media;

means for transmitting the data and the streaming media from a server to a client over a network;

means for integrating the streaming media and the simulator data for playback in a single web browser; and

means for synchronously playing the streaming media with the data in the single web browser on the client,

wherein the means for integrating links the streaming media with the simulator data such that the streaming media and the simulator data are both displayed with reference to a timeline and such that when moving to a particular point on the timeline, corresponding streaming media and simulator data are played by the means for synchronously playing from said particular point on the timeline.