Multi-perspective viewer for content-based interactivity

• US 6,144,375 A
• Filed: 08/14/1998
• Issued: 11/07/2000
• Est. Priority Date: 08/14/1998
• Status: Expired due to Term

• Current Assignee: Cloud Software Group, Eastman Kodak Company
• Original Assignee: PRAJA, Inc. (Cloud Software Group)
• Inventors: Kuramura, Don Yamato, Katkere, Arun L., Bailey, Asquith A., McKinley, Ryan B., Hicks, Terry Randolph, Jain, Ramesh C.
• Primary Examiner(s): Feild, Joseph H.
• Assistant Examiner(s): Bourque, Robert D.
• Application Number:
  US09/134,188
• Time in Patent Office: 816 Days
• Field of Search: 707/530, 707/521, 707/5, 707/10, 707/102, 707/104, 345/302, 345/328, 345/334, 345/355
• US Class Current: 715/251
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and apparatus for interactively viewing a real-world environment. The viewer includes a user interface having a first window for displaying a two-dimensional representation of a three-dimensional model of the real world environment. The viewer further includes a plurality of other viewing areas for displaying and querying for views of the real-world environment. The viewer includes a content-based event timeline that graphically depicts multi-media events satisfying user queries. Several methods can be used to select an event for display. For example, the user can select an event by selecting the event representation from the timeline. Alternatively, the user can select an event by querying the system for selected objects, viewing angles, input devices, etc. The viewer synchronizes and links together all of the multi-media data types associated with a selected event. Thus, when the user selects an event, the viewer displays all of the multi-media information (such as audio/video and textual information) that is associated with the selected event.

Claims

1. A multi-perspective viewer for interactively viewing a multi-media program derived from a real-world environment, the multi-media program comprising a plurality of multi-media events represented by associated and corresponding multiple multi-media data types stored in a multi-media database, comprising:

a. user interface means for accepting input from a user and for outputting information from the viewer to the user, wherein the user interface means includes at least one input device and a visual display;

b. query means for querying the viewer for selected multi-media events, wherein the query means displays an event timeline on the visual display that graphically depicts events that satisfy a selected query;

c. selecting means, responsive to input from the user interface means, for selecting an event from the event timeline; and

d. display means, responsive to the selecting means, for displaying the multi-media data types associated with the selected event on the visual display.

2. The multi-perspective viewer of claim 1, wherein the database is created by filtering the multi-media program and capturing only the multi-media events that satisfy event selection criteria.

3. The multi-perspective viewer of claim 2, wherein the database is a spatial/temporal database and wherein the multiple multi-media data types are indexed for subsequent retrieval by query means.

4. The multi-perspective viewer of claim 3, wherein the database is a relational database.

5. The multi-perspective viewer of claim 3, wherein the database is object-oriented.

6. The multi-perspective viewer of claim 2 wherein the event selection criteria is specific to a particular multi-media program.

7. The multi-perspective viewer of claim 2 wherein the event selection criteria is determined by a personality module having pre-defined knowledge of the multi-media program processed by the viewer.

8. The multi-perspective viewer of claim 7, wherein the personality module comprises a utility process capable of identifying significant multi-media events.

9. The multi-perspective viewer of claim 2, wherein the multi-media event selection criteria include inputs from a system user.

10. The multi-perspective viewer of claim 9, wherein the system user inputs include annotation data and control information.

11. The multi-perspective viewer of claim 10, wherein the system user inputs include event starting and event ending control inputs.

12. The multi-perspective viewer of claim 10, wherein the system user inputs include delete, save and override control inputs.

13. The multi-perspective viewer of claim 3, wherein the multi-media database includes fully synchronized audio, video, statistical data, and other data corresponding to the plurality of mutli-media events that satisfy the event selection criteria.

14. The multi-perspective viewer of claim 13, wherein the multi-media database is stored in a hot-swappable audio-video hard disk.

15. The multi-perspective viewer of claim 1, wherein the display means controls the display of the following windows on the visual display:

a. a video window capable of displaying real-time video clips of the selected event;

b. a user control window whereby the user can control the display of information on the visual display;

c. a 2D model window capable of displaying a 3D model of the selected event displayed in the video window;

d. a model navigation control window that allows the user to control the display of video information in the video window and the display of the 3D model in the model window; and

e. an event timeline window for displaying the event timeline.

16. The multi-perspective viewer of claim 15, wherein the display means further controls the display of the following windows on the visual display:

a. an audio control window;

b. a bookmark bin window;

c. a context-sensitive VCR control window;

d. a time code window;

e. a camera angle control window;

f. a statistics and textual information viewing window; and

g. a query processing window.

17. The multi-perspective viewer of claim 15, wherein the video clips displayed in the video window are encoded and compressed prior to being stored in the multi-media database.

18. The multi-perspective viewer of claim 15, wherein the user control window includes the following user control buttons:

a. a swap control button for which allows the user to interchange the positions of the video window and the model window on the visual display;

b. a freeze-frame control button which allows the user to freeze the real-time video clip displayed in the video window;

c. a bookmarks control button for creating user bookmarks;

d. a preferences pop-up control button;

e. a zoom control button which allows the user to zoom in and out on an image displayed by the real-time video clip; and

f. a hyperlink control button.

19. The multi-perspective viewer of claim 15, wherein the model window comprises a 2D representation of a 3D environmental model of the real-world environment.

20. The multi-perspective viewer of claim 19, wherein the 3D environmental model comprises a spatial-temporal database representing a sensed real-world environment, and wherein the spatial coordinates of the 3D model approximately correspond to the spatial coordinates of the sensed real-world environment, and wherein objects in the 2D model window are linked to objects in the 3D model.

21. The multi-perspective viewer of claim 15, wherein the model navigation control window includes context-sensitive control buttons that allow the user to control the display of selected multi-media events on the visual display.

22. The multi-perspective viewer of claim 21, wherein the context-sensitive control buttons include an object tracking button that allows the user to track the progress of a selected object throughout a selected multi-media event.

23. The multi-perspective viewer of claim 15 wherein the event timeline comprises a two-dimensional representation of the synchronized and linked multi-media data types in the multi-media database.

24. The multi-perspective viewer of claim 23, wherein the event timeline is synchronized and linked to the display means and thereby the display of every window on the visual display, and wherein the user selects a desired multi-media event by interacting with the event timeline and thereby causing the display means to correspondingly update each of the windows on the visual display.

25. The multi-perspective viewer of claim 24, wherein the event timeline changes in response to the query means.

26. The multi-perspective viewer of claim 16, wherein the query processing window includes a plurality of pop-up menus for formulating complex event queries.

27. The multi-perspective viewer of claim 16, wherein the audio control window includes controls for controlling volume, muting, mono/stereo, and selection of audio sources.

28. The multi-perspective viewer of claim 26, wherein the plurality of pop-up menus includes an object pop-up menu including an object profile.

29. The multi-perspective viewer of claim 1, wherein the user is a sports coach, and wherein the viewer displays information on the visual display in response to input provided by the coach.

30. The multi-perspective viewer of claim 16, wherein the context-sensitive VCR control window includes a smart fast forward and reverse control button for forwarding and reversing the display of subsequent and previous events, respectively.

31. A method for interactively displaying a multi-media program derived from a real-world environment, the multi-media program comprising a plurality of multi-media events represented by associated and corresponding multiple multi-media data types stored in a multi-media database, the method comprising the steps of:

a. accepting a query wherein the query specifies selected multi-media events;

b. displaying an event timeline on a visual display wherein the event timeline graphically depicts events that satisfy the query;

c. selecting an event from the event timeline; and

d. displaying the multi-media data types associated with the selected event on the visual display.

32. The method of claim 31, further comprising the step of outputting audio, textual and other data associated with the multi-media data types displayed in step (d).

33. A system for interactively viewing a multi-media program derived from a real-world environment, comprising:

a. a computer environment including an input device and a visual display;

b. a plurality of sensors connected to the computer environment and sensing the real-world environment;

c. capturing means executed by the computer environment for capturing a plurality of multi-media events sensed by the sensors;

d. filtering means executed by the computer environment for filtering the plurality of multi-media events captured by the capturing means, and for producing a multi-media program, wherein the multi-media program includes only those multi-media events that satisfy event selection criteria;

e. an environmental model stored in the computer environment, wherein the environmental model represents the multi-media program, and wherein the spatial coordinates of the environmental model approximately correspond to the spatial coordinates of the sensed real-world environment; and

f. a viewer for interactively accessing selected multi-media events and objects from the multi-media program, wherein the viewer includes an event timeline that graphically displays events stored in the computer environment.

34. The system for interactively viewing a multi-media program as defined in claim 33, wherein the sensors comprise a single video camera and a plurality of sources capable of producing non-video information related to the multi-media program.