Event based organization and access of digital photos

• US 8,732,161 B2
• Filed: 04/25/2008
• Issued: 05/20/2014
• Est. Priority Date: 04/27/2007
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Jain, Ramesh, Westermann, Gerd Utz, Scherp, Ansgar, Agaram, Srikanth
• Primary Examiner(s): Hwa, Shyue Jiunn
• Application Number:
  US12/110,010
• Publication Number:
  US 20090319472A1
• Time in Patent Office: 2,216 Days
• Field of Search: 707/716, 707/713, 707/715
• US Class Current: 707/716
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Architecture and its implementation of an event-based media management system and method are disclosed. A generic model for events is used for developing architecture for media management, called Eventcentric Media Management (EMMa) system. The system focuses on event experiences and provides flexible functionality for media management. The system and method analyze cross-media files for features to extract atomic events and apply algorithms to cluster files including similar or related features. Thus, the EMMa system allows unified media indexing of multiple media types, better abstraction of the real world, use of context, media processing operations to compute and store features and tags, and a more intuitive support for multiple users. It provides environments for querying and browsing media related to events and authoring of multimedia presentations for sharing with other individuals.

Claims

1. A method of cataloguing media files for event-centric organization, comprising:

scanning media files for atomic events;

applying feature extraction techniques to the atomic events to obtain context information and content information for each atomic event;

classifying the atomic events into predetermined classes based on the extracted context and content information;

presenting, to a user, the classified atomic events for a user'"'"'s ratification or rejection;

assembling composite events from the ratified atomic events;

querying for retrieval of qualified ones of the assembled composite events;

assigning an event type to the assembled composite events; and

populating parameters defined for the assigned event type with values extracted from atomic events of the assembled composite events;

comparing time difference and dissimilarity between two successive events and combining them together; and

automatically creating clusters of the atomic events with a plurality of levels of granularity.

2. The method of cataloguing media files for event-centric organization claim 1, wherein:

the context and content information includes assigning metadata tags to the atomic events; and

the assembling of composite events uses the metadata tags.

3. The method of cataloguing media files for event-centric organization of claim 2, wherein:

the metadata tags are selected from a group of dimensions including time, location, persons involved, displayed items, and activities.

4. The method of cataloguing media files for event-centric organization claim 1, further comprising:

presenting the media files in an electronic medium displayed as the assembled composite events.

5. The method of cataloguing media files for event-centric organization claim 1, wherein:

the media files are photos including EXIF headers containing context information; and

the content information extracted includes color histogram and texture data.

6. The method of cataloguing media files for event-centric organization claim 1, further comprising:

a tagging process employing the manual addition of keywords to the atomic events.

7. The method of cataloguing media files for event-centric organization claim 1, wherein:

the assembling composite events are performed manually, semi-automatically, or automatically, and wherein the event type is automatically assigned using characteristics of the atomic events of the assembled composite events.

8. The method of cataloguing media files for event-centric organization claim 1, further comprising:

grouping atomic events into composite events using color histogram and visual characteristic event detection of the media files scanning for and comparing time and visual information; and

arranging media files of the atomic events based on a structure of the assembled composite events.

9. The method of cataloguing media files for event-centric organization claim 8, wherein:

events are clustered based on global positioning satellite information.

10. The method of cataloguing media files for event-centric organization claim 8, further comprising:

applying the clustering process to media files of the assembled composite events for determining composite events within the assembled composite events; and

providing an interface for the user to browse through the assembled composite events;

wherein the media files include video data and events are clustered based on facial detection information.

11. A system for organizing images comprising:

a server in communication with a network, the server having an event base database, a user database, a file system, at least three wrappers for abstracting data from at least one of the event base database, the user database, and the file system, and the server having a service layer for storing atomic events and composite events, retrieving atomic or composite events, deleting atomic or composite events, and updating atomic or composite events; and

a client in communication with the network, the client having an application layer for programming an interface for the client to access event data and media data storage, perform event digestion, and perform event annotation, wherein the client periodically receives user modifications to the composite events, and wherein the application layer includes a query function for retrieval of qualified composite events.

12. The system for organizing images of claim 11, wherein:

the event base database, user database, and file system employ different media storage mechanisms for optimized data storage and retrieval of different types.

13. The system for organizing images of claim 11, wherein:

the service layer includes a function for organizing images according to event relationships.

14. A method for using an event-centric management architecture to retrieve media data, the method comprising:

analyzing media data from at least one source;

organizing and storing the media data according to events including atomic events into composite events using a library of dimensional tags associated to each of the atomic events;

assigning an event type to the composite events;

annotating the composite events with parameters defined for the event type;

accessing a collection of composite events for display and browsing on a graphical user interface using an initial graphical presentation;

performing a query-based search on the graphical user interface using predetermined dimensional tags in the library to search for selected composite events;

switching from the initial graphical presentation to a second graphical presentation to display the selected composite events based on one of the dimensional tags in the library;

using the events to retrieve media data;

displaying the composite events as polygons that map the spatial extent of the composite events and contain atomic events as icons within the polygons; and

displaying a timeline of the composite events with event details and event media assets displayed above the timeline.

15. The method for using the event-centric management architecture to retrieve digital images of claim 14, wherein:

the media data are organized according to composite events and include metadata tags describing the activity of said events.

16. The method for using the event-centric management architecture to retrieve digital images of claim 15, further comprising:

presenting the media data in a multimedia format.

17. The method for using the event-centric management architecture to retrieve digital images of claim 16, further comprising:

grouping events into multimedia albums for event browsing.