Concurrent voice to text and sketch processing with synchronized replay

• US 20040193428A1
• Filed: 04/13/2004
• Published: 09/30/2004
• Est. Priority Date: 05/12/1999
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Fruchter, Renate, Biswas, Pratik
• Granted Patent: US 7,458,013 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

The present invention facilitates transparent and cost effective capture, sharing, and re-use of knowledge in informal media such as audio, video, and individual sketching actions. Sketches, handwritings, verbal comments and the likes captured during a session are timestamped and, once completed, automatically indexed and published on a web server that allows for real time, streamed and synchronized replay thereof. All timestamps are converted to a common time base. To control the synchronous replay, multiple treads collaborate based on common base timestamps corresponding to the sketching actions and correlated audio/video stream(s). An enhanced interactive graphical user interface enables a network user to create and capture a session, e.g., annotate an imported image with sketches and verbal comments. The user or others can access the published session in real time and select a keyword or an individual sketch entity from which to begin replay. Transcribed audio stream can also be synchronously replayed.

Claims

1. A method for capturing, synchronizing, and replaying a sketching activity and media information associated to said sketching activity, said method comprising:

a) simultaneously timestamped capturing said sketching activity and said media information;

wherein

said sketching activity producing one or more sketch objects, each having a corresponding sketch object timestamp, wherein

said media information includes audio data, and wherein

said audio data are captured in an audio file;

b) transcribing said audio file, recording respectively recognized keywords or phrases and their corresponding timestamps;

c) converting all timestamps associated to said one or more sketch objects, said audio file, and said keywords or phrases to a common time base;

d) enabling a user to select a starting point for replay, said starting point is one or more of said sketch objects, a keyword, or a phrase;

e) based on said starting point, synchronizing said one or more sketch objects, said audio file, and said keywords or phrases, utilizing their respective corresponding common base timestamps; and

f) replaying said sketching activity and said media information based on said starting point.

2. The method according to claim 1, wherein

sketch object timestamp=raw sketch object timestamp−

session start timestamp;

audio timestamp=raw audio timestamp*1000;

keyword timestamp=system clock keyword timestamp−

session start timestamp; and

transcribed data timestamp=(Tr*Ds/Dr)+Tsst, where

Tr=raw transcribed data timestamp−

raw start time,

Ds=system clock session end time−

system clock session start time,

Dr=raw end time−

raw start time, and

Tsst=system clock applet start time.

3. The method according to claim 2, further comprising:

comparing said sketch object timestamp with a keyword timestamp associated with said starting point; and

replaying said sketch activity and said media information starting from the latest sketch object drawn before said keyword was spoken.

4. The method according to claim 3, further comprising:

synchronously displaying text corresponding to said audio data, synchronously playing video data corresponding to said sketching activity, synchronously playing video data corresponding to said audio data, or a combination thereof.

5. The method according to claim 1, further comprising:

importing a background image of which said sketch activity annotates.

6. The method according to claim 1, further comprising:

automatically indexing and storing said sketch activity and said media information in a database.

7. The method according to claim 6, further comprising:

distributing via real time Internet streaming said sketch activity and said media information over a computer network;

wherein said timestamped capturing occurs at a first computer connected thereto and said replaying occurs at a second computer connected thereto.

8. The method according to claim 7, further comprising:

enabling a user of said second computer access to said sketch activity and said media information via an interactive graphical user interface.

9. The method according to claim 8, wherein

said interactive graphical user interface and said database are maintained by a server connected to said computer network.

10. The method according to claim 7, wherein

said computer network is characterized as an intranet, the Internet, or a combination thereof, said computer network comprising wired and wireless communication links.

11. A digital computer system programmed to perform the method of claim 1.

12. A computer-readable medium storing a computer program implementing the method of claim 1.

13. A system for real time capturing and synchronous replaying a sketching activity and related media information, said system comprising:

means for capturing said sketching activity and said media information simultaneously;

wherein said media information includes audio signals and video signals and said sketching activity producing a plurality of sketch entities;

means for encoding said sketch entities with corresponding sketch object timestamps and said media information with corresponding media timestamps;

means for converting said sketch object timestamps to sketch object common base timestamps and said media timestamps to media common base timestamps;

means for comparing said sketch object common base timestamps and said media common base timestamps; and

an interactive graphical user interface for synchronously replaying said sketch activity and said media information;

14. The system of claim 13, further comprises:

means for transcribing said audio signals into recognizable words and phrases;

means for associating said recognizable words and phrases with corresponding keyword timestamps; and

means for converting said keyword timestamps to keyword common base timestamps.

15. The system of claim 13, further comprises:

means for enabling a user to select a keyword from said recognizable words and phrases; and

means for comparing said sketch object common base timestamps with a keyword common base timestamp associated with said selected keyword and determining a starting point from when said interactive graphical user interface replays simultaneously and in synchronization said sketch activity, said media information, and a text representation of said media information.

16. The system of claim 13, further comprising:

a database for storing said sketch activity and said media information;

means for distributing said sketch activity and said media information over a computer network;

means for enabling one or more user units distributed in said computer network to modify said sketch activity and said media information; and

means for updating said database in accordance with said modification.

17. The system of claim 16, further comprising:

a server for maintaining said database and said interactive graphical user interface.

18. A computer-readable medium storing a computer program implementing the system of claim 13, wherein said interactive graphical user interface comprises a color pallet, a virtual canvas, and a plurality of functionalities.

19. The computer-readable medium of claim 18, wherein said functionalities comprise:

means for removing the last drawing movement from said canvas;

means for removing all drawing movements on said canvas;

means for retrieving and importing a background image onto said canvas;

means for removing said background image;

means for retrieving, importing and displaying slides;

means for placing a 2 dimensional or a 3 dimensional grid on said canvas;

means for storing said sketch activity as a subpage;

means for importing said subpage onto said canvas;

means for importing drawing movements of said subpage onto said canvas;

means for tracing said drawing movements in a buffer;

means for placing content of said buffer onto said canvas;

means for taking a screenshot of said canvas; and

means for placing said screenshot on said canvas as background.

20. The computer-readable medium of claim 19, wherein each of said functionalities is activated and deactivated via a voice command or a screen representation characterized as a button.