Concurrent voice to text and sketch processing with synchronized replay

• US 7,458,013 B2
• Filed: 04/13/2004
• Issued: 11/25/2008
• Est. Priority Date: 05/12/1999
• Status: Expired due to Term

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Fruchter, Renate, Biswas, Pratik
• Primary Examiner(s): Campbell; Joshua D
• Application Number:
  US10/824,063
• Publication Number:
  US 20040193428A1
• Time in Patent Office: 1,687 Days
• Field of Search: 715/500, 715/500.1, 715/501.1, 715/502, 715/512, 715/200, 715/201, 715/202, 715/203, 715/205, 715/230, 715/233
• US Class Current: 715/203
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

The present invention facilitates transparent and cost effective capture, sharing, and re-use of knowledge in informal media such as audio, video, and individual sketching actions. Sketches, handwritings, verbal comments and the likes captured during a session are timestamped and, once completed, automatically indexed and published on a web server that allows for real time, streamed and synchronized replay thereof. All timestamps are converted to a common time base. To control the synchronous replay, multiple treads collaborate based on common base timestamps corresponding to the sketching actions and correlated audio/video stream(s). An enhanced interactive graphical user interface enables a network user to create and capture a session, e.g., annotate an imported image with sketches and verbal comments. The user or others can access the published session in real time and select a keyword or an individual sketch entity from which to begin replay. Transcribed audio stream can also be synchronously replayed.

Claims

1. A method for capturing, synchronizing, and replaying a sketching activity and media information associated to said sketching activity, said method comprising:

a) simultaneously raw timestamped capturing said sketching activity and said media information;

wherein

said sketching activity producing one or more sketch objects, each having a corresponding raw sketch object timestamp, wherein

said media information includes audio data, and wherein

said audio data are captured in an audio file;

b) transcribing said audio file, recording respectively recognized keywords or phrases and their corresponding raw timestamps;

c) converting all raw timestamps associated to said one or more sketch objects, said audio file, and said keywords or phrases to common base timestamps;

wherein

said converting includes scaling raw timestamps to a common unit of time and offsetting said scaled timestamps to a same initial time, and wherein

said converting establishes at least a sketch object common base timestamp and a keyword common base timestamp;

d) enabling a user to select a starting point for replay, said starting point is one or more of said sketch objects, said keywords, or said phrases;

e) based on said starting point, synchronizing said one or more sketch objects, said audio file, and said keywords or phrases, utilizing their respective corresponding common base timestamps;

f) replaying said sketching activity and said media information based on said starting point; and

g) storing a session start timestamp, an session end timestamp, a raw start time, and a raw end time, wherein

said sketch object common base timestamp=said raw sketch object timestamp−

said session start timestamp;

said transcribed data common base timestamp=(Tr*Ds/Dr)+Tsst, wherein

Tr=said raw transcribed data timestamp−

said raw start time,

Ds=said session end timestamp−

said session start timestamp,

Dr=said raw end time−

said raw start time, and

Tsst=said session start timestamp; and

wherein

said keyword common base timestamp=a system clock keyword timestamp−

said session start timestamp.

2. The method according to claim 1, further comprising:

comparing said sketch object common base timestamp with said keyword common base timestamp associated with said starting point; and

replaying said sketch activity and said media information starting from the latest sketch object drawn before said keyword was spoken.

3. The method according to claim 1, further comprising:

synchronously displaying text corresponding to said audio data, synchronously playing video data corresponding to said sketching activity, synchronously playing video data corresponding to said audio data, or a combination thereof.

4. The method according to claim 1, further comprising:

importing a background image of which said sketch activity annotates.

5. The method according to claim 1, further comprising:

automatically indexing and storing said sketch activity and said media information in a database.

6. The method according to claim 5, further comprising:

distributing via real time streaming said sketch activity and said media information over a computer network;

wherein said timestamped capturing occurs at a first computer connected thereto and said replaying occurs at a second computer connected thereto.

7. The method according to claim 6, further comprising:

enabling a user of said second computer access to said sketch activity and said media information via an interactive graphical user interface.

8. The method according to claim 7, wherein

said interactive graphical user interface and said database are maintained by a server connected to said computer network.

9. The method according to claim 6, wherein

said computer network is characterized as an intranet, the Internet, or a combination thereof, said computer network comprising wired and wireless communication links.

10. A digital computer system programmed to perform the method of claim 1.

11. A computer-readable medium storing a computer program implementing the method of claim 1.