DiVAS-a cross-media system for ubiquitous gesture-discourse-sketch knowledge capture and reuse

• US 20050283752A1
• Filed: 05/17/2005
• Published: 12/22/2005
• Est. Priority Date: 05/17/2004
• Status: Abandoned Application

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Yin, Zhen, Fruchter, Renate, Biswas, Pratik
• Application Number:
  US11/132,171
• Publication Number:
  US 20050283752A1
• Time in Patent Office: Days
• US Class Current: 717/100
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The invention provides a cross-media software environment that enables seamless transformation of analog activities, such as gesture language, verbal discourse, and sketching, into integrated digital video-audio-sketching (DiVAS) for real-time knowledge capture, and that supports knowledge reuse through contextual content understanding.

Claims

1. A method of processing a video stream, comprising the step of:

enabling a user to define a scenario-specific gesture vocabulary database over selected segments of said video stream having an object performing gestures; and

according to said gesture vocabulary, automatically identifying gestures and their corresponding time of occurrence from said video stream.

2. The method according to claim 1, further comprising:

extracting said object from each frame of said video stream;

classifying state of said extracted object in each said frame; and

analyzing sequences of states to identify actions being performed by said object.

3. The method according to claim 2, further comprising:

determining a contour or shape of said object.

4. The method according to claim 2, further comprising:

determining a skeleton of said object.

5. The method according to claim 1, further comprising:

encoding said video stream into a predetermined format.

6. The method according to claim 1, further comprising:

enabling said user to specify a transition matrix that identifies transition costs between states.

7. The method according to claim 6, further comprising:

finding a minimum cost path over said transition matrix.

8. A computer system programmed to implement the method steps of claim 1.

9. A program storage device accessible by a computer, tangibly embodying a program of instructions executable by said computer to perform the method steps of claim 1.

10. A cross-media system, comprising:

an information retrieval analysis subsystem for adding structure to and retrieving information from unstructured speech transcripts;

a video analysis subsystem for enabling a user to define a scenario-specific gesture vocabulary database over selected segments of a video stream having an object performing gestures and for identifying gestures and their corresponding time of occurrence from said video stream.;

an audio analysis subsystem for capture and reusing verbal-discourse; and

a sketch analysis subsystem for capturing, indexing, and replaying audio and sketch.