Method and apparatus for retrieving multimedia data through spatio-temporal activity maps

• US 20030023595A1
• Filed: 06/12/2002
• Published: 01/30/2003
• Est. Priority Date: 10/06/1998
• Status: Active Grant

• Current Assignee: Nokia of America Corporation (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Pingali, Gopal S., Opalach, Agata, Carlbom, Ingrid Birgitta
• Granted Patent: US 7,143,083 B2
• Time in Patent Office: Days
• US Class Current: 707/7
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06F 2216/01 Automatic library building
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Techniques for retrieval of multimedia data through visual representations are provided. Such visual representations, preferably in the form of visual activity maps or spatio-temporal activity maps, serve as an efficient and intuitive graphical user interface for multimedia retrieval, particularly when the media streams are derived from multiple sensors observing a physical environment. An architecture for interactive media retrieval is also provided by combining such visual activity maps with domain specific event information. Visual activity maps are derived from the trajectories of motion of objects in the environment. The visual activity map based techniques significantly help users in quickly and effectively discovering interesting portions of the data, and randomly accessing and retrieving the corresponding portions of the media streams.

Claims

1. A method of performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the method comprising the steps of:

generating at least one visual representation of an activity function related to at least one activity associated with the monitored application, in response to an input from a user;

selecting, by the user, a desired aspect of the visual representation; and

retrieving, for presentation to the user, at least a portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user.

2. The method of claim 1, further comprising the step of generating at least one refined visual representation, before the retrieval step, in response to the aspect selection step.

3. The method of claim 1, wherein the user input comprises one or more of a spatial region associated with the visual representation, a temporal region associated with the visual representation, a spatio-temporal region associated with the visual representation, specification of at least one event associated with the application, and an activity function selection.

4. The method of claim 1, wherein the visual representation is a visualization of a geometry of the environment and of a spatially distributed activity in the environment over a particular time period.

5. The method of claim 1, whereby the portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user is randomly accessed and retrieved from the database.

6. The method of claim 1, wherein the visual representation is in the form of an activity map.

7. The method of claim 1, wherein the visual representation is at least partially derived from trajectories of the motion of one or more objects in the environment.

8. The method of claim 1, wherein the visual representation represents at least a partial summarization of media associated with the application.

9. The method of claim 1, wherein the application is a real time application.

10. The method of claim 1, wherein the application is one of a security-related application, a surveillance-related application and a sporting application.

11. The method of claim 1, wherein the generating step further comprises generating at least one index structure for at least one cell associated with the environment.

12. Apparatus for performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the apparatus comprising:

at least one processor operative to;

(i) generate at least one visual representation of an activity function related to at least one activity associated with the monitored application, in response to an input from a user;

(ii) enable selection, by the user, of a desired aspect of the visual representation; and

(iii) retrieve, for presentation to the user, at least a portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user.

13. The apparatus of claim 12, wherein the at least one processor is further operative to generate at least one refined visual representation, before the retrieval operation, in response to the aspect selection operation.

14. The apparatus of claim 12, wherein the user input comprises one or more of a spatial region associated with the visual representation, a temporal region associated with the visual representation, a spatio-temporal region associated with the visual representation, specification of at least one event associated with the application, and an activity function selection.

15. The apparatus of claim 12, wherein the visual representation is a visualization of a geometry of the environment and of a spatially distributed activity in the environment over a particular time period.

16. The apparatus of claim 12, whereby the portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user is randomly accessed and retrieved from the database.

17. The apparatus of claim 12, wherein the visual representation is in the form of an activity map.

18. The apparatus of claim 12, wherein the visual representation is at least partially derived from trajectories of the motion of one or more objects in the environment.

19. The apparatus of claim 12, wherein the visual representation represents at least a partial summarization of media associated with the application.

20. The apparatus of claim 12, wherein the generating operation further comprises generating at least one index structure for at least one cell associated with the environment.

21. An article of manufacture for performing data retrieval in accordance with a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the article comprising a machine readable medium containing one or more programs which when executed implement the steps of:

generating at least one visual representation of an activity function related to at least one activity associated with the monitored application, in response to an input from a user;

selecting, by the user, a desired aspect of the visual representation; and

retrieving, for presentation to the user, at least a portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user.

22. A system for providing an interface to a database storing media associated with an application and captured by one or more sensors monitoring an environment associated with the application, the system comprising:

a visual representation generator, operatively coupled to the database; and

a graphical user interface, operatively coupled to the visual representation generator and the database;

wherein the visual representation generator generates at least one activity map associated with the monitored application, in response to an input from a user;

the graphical user interface presents the activity map to the user;

the user selects a desired aspect associated with the activity map via the graphical user interface; and

the graphical user interface retrieves, for presentation to the user, at least a portion of the media stored in the database that corresponds to the desired aspect of the visual representation selected by the user.

23. The system of claim 22, wherein the visual representation generator generates at least one refined visual representation for presentation to the user, before media retrieval, in response to the aspect selected by the user.

24. The system of claim 22, wherein the visual representation generator comprises a filter for querying the database to retrieve trajectories of the motion of one or more objects in the environment corresponding to at least one event and at least one time period of the application, as specified by the user.

25. The system of claim 22, wherein the visual representation generator comprises a spatial mapping module for mapping object trajectories.

26. The system of claim 22, wherein the visual representation generator comprises a filter for filtering object trajectories based on a spatial selection specified by the user.

27. The system of claim 22, wherein the visual representation generator comprises an aggregator for generating the activity map based on a mapping selection specified by the user.

28. The system of claim 22, wherein the graphical user interface comprises a map navigator module for creating a presentation of the activity map for the user based on data associated with the activity map and a geometry associated with the environment.

29. The system of claim 22, wherein the graphical user interface comprises a media browser for retrieving relevant media from the database.