Method and apparatus for retrieving multimedia data through spatio-temporal activity maps

• US 7,143,083 B2
• Filed: 06/12/2002
• Issued: 11/28/2006
• Est. Priority Date: 06/12/2001
• Status: Expired due to Term

• Current Assignee: Nokia of America Corporation (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Carlbom, Ingrid Birgitta, Opalach, Agata, Pingali, Gopal S.
• Primary Examiner(s): Coby, Frantz
• Application Number:
  US10/167,539
• Publication Number:
  US 20030023595A1
• Time in Patent Office: 1,630 Days
• Field of Search: 707/2, 707/6, 707/102, 707/3, 707/4, 707/5, 345/530
• US Class Current: 1/1
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06F 2216/01 Automatic library building
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Techniques for retrieval of multimedia data through visual representations are provided. Such visual representations, preferably in the form of visual activity maps or spatio-temporal activity maps, serve as an efficient and intuitive graphical user interface for multimedia retrieval, particularly when the media streams are derived from multiple sensors observing a physical environment. An architecture for interactive media retrieval is also provided by combining such visual activity maps with domain specific event information. Visual activity maps are derived from the trajectories of motion of objects in the environment. The visual activity map based techniques significantly help users in quickly and effectively discovering interesting portions of the data, and randomly accessing and retrieving the corresponding portions of the media streams.

Claims

1. A method of performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the method comprising the steps of:

computing an activity function related to at least one activity associated with the monitored application;

computing at least one index to the media stored in the database that corresponds to the computed activity function;

generating at least one visual representation of the activity function, in response to an input event, wherein the visual representation is in the form of an activity map;

selecting, in response to the input event or another input event, a desired aspect of the activity map; and

retrieving via the computed index, for presentation, at least a portion of the media stored in the database that corresponds to the desired aspect of the activity map.

2. The method of claim 1, further comprising the step of generating at least one refined visual representation, before the retrieval step, in response to the aspect selection step.

3. The method of claim 1, wherein at least one of the input events comprises one or more of a spatial region associated with the visual representation, a temporal region associated with the visual representation, a spatio-temporal region associated with the visual representation, specification of at least one event associated with the application, and an activity function selection.

4. The method of claim 1, wherein the visual representation is a visualization of a geometry of the environment and of a spatially distributed activity in the environment over a particular time period.

5. The method of claim 1, whereby the portion of the media stored in the database that corresponds to the desired aspect of the visual representation is randomly accessed and retrieved from the database.

6. The method of claim 1, wherein the visual representation is at least partially derived from trajectories of the motion of one or more objects in the environment.

7. The method of claim 1, wherein the visual representation represents at least a partial summarization of media associated with the application.

8. The method of claim 1, wherein the application is a real time application.

9. The method of claim 1, wherein the application is one of a security-related application, a surveillance-related application and a sporting application.

26. The method of claim 1, wherein the computed activity function comprises a spatial coverage activity function representative of a spatial coverage associated with an object being monitored in the environment.

27. The method of claim 1, wherein the computed activity function comprises a speed activity function representative of a speed associated with an object being monitored in the environment.

28. The method of claim 1, wherein the computed activity function comprises a result of an integral operator or a differential operator applied to object tracking information in the environment.

10. Apparatus for performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the apparatus comprising:

at least one processor operative to;

(i) compute an activity function related to at least one activity associated with the monitored application;

(ii) compute at least one index to the media stored in the database that corresponds to the computed activity function;

(iii) generate at least one visual representation of the activity function, in response to an input event, wherein the visual representation is in the form of an activity map;

(iv) enable selection, in response to the input event or another input event, of a desired aspect of the activity map; and

(v) retrieve via the computed index, for presentation, at least a portion of the media stored in the database that corresponds to the desired aspect of the activity map.

11. The apparatus of claim 10, wherein the at least one processor is further operative to generate at least one refined visual representation, before the retrieval operation, in response to the aspect selection operation.

12. The apparatus of claim 10, wherein at least one of the input events comprises one or more of a spatial region associated with the visual representation, a temporal region associated with the visual representation, a spatio-temporal region associated with the visual representation, specification of at least one event associated with the application, and an activity function selection.

13. The apparatus of claim 10, wherein the visual representation is a visualization of a geometry of the environment and of a spatially distributed activity in the environment over a particular time period.

14. The apparatus of claim 10, whereby the portion of the media stored in the database that corresponds to the desired aspect of the visual representation is randomly accessed and retrieved from the database.

15. The apparatus of claim 10, wherein the visual representation is at least partially derived from trajectories of the motion of one or more objects in the environment.

16. The apparatus of claim 10, wherein the visual representation represents at least a partial summarization of media associated with the application.

17. An article of manufacture for performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the article comprising a machine readable recording medium containing one or more programs which when executed implement the steps of:

computing an activity function related to at least one activity associated with the monitored application;

computing at least one index to the media stored in the database that corresponds to the computed activity function;

generating at least one visual representation of the activity function, in response to an input event, wherein the visual representation is in the form of an activity map;

selecting, in response to the input event or another input event, a desired aspect of the activity map; and

retrieving via the computed index, for presentation, at least a portion of the media stored in the database that corresponds to the desired aspect of the activity map.

18. A system for providing an interface to a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the system comprising:

a visual representation generator, operatively coupled to the database; and

a graphical user interface, operatively coupled to the visual representation generator and the database;

wherein the visual representation generator generates at least one activity map associated with the monitored application, in response to an input event;

the graphical user interface presents the activity map;

a desired aspect associated with the activity map is selected via the graphical user interface; and

the graphical user interface retrieves, for presentation, at least a portion of the media stored in the database that corresponds to the desired aspect of the visual representation;

further wherein the activity map corresponds to at least one activity function computed for at least one activity associated with the monitored application, and media is retrieved from the database via at least one corresponding index computed for the at least one activity function.

19. The system of claim 18, wherein the visual representation generator generates at least one refined visual representation for presentation, before media retrieval, in response to the aspect selected.

20. The system of claim 18, wherein the visual representation generator comprises a filter for querying the database to retrieve trajectories of the motion of one or more objects in the environment corresponding to at least one event and at least one time period of the application.

21. The system of claim 18, wherein the visual representation generator comprises a spatial mapping module for mapping object trajectories.

22. The system of claim 18, wherein the visual representation generator comprises a filter for filtering object trajectories based on a spatial selection.

23. The system of claim 18, wherein the visual representation generator comprises an aggregator for generating the activity map based on a mapping selection.

24. The system of claim 18, wherein the graphical user interface comprises a map navigator module for creating a presentation of the activity map based on data associated with the activity map and a geometry associated with the environment.

25. The system of claim 18, wherein the graphical user interface comprises a media browser for retrieving relevant media from the database.