Method for event-based semantic classification

• US 8,611,677 B2
• Filed: 11/19/2008
• Issued: 12/17/2013
• Est. Priority Date: 11/19/2008
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Intellectual Ventures Fund 83 LLC (Intellectual Ventures LLC)
• Inventors: Das, Madirakshi, Loui, Alexander C., Wood, Mark D.
• Primary Examiner(s): Chu, Randolph I
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US12/273,600
• Publication Number:
  US 20100124378A1
• Time in Patent Office: 1,854 Days
• Field of Search: 382/100, 382/162, 382/168, 382/181, 382/190, 382224-228, 382/278, 382/305, 382/155, 382/159, 382/195, 707821-831, 715/200, 715/240
• US Class Current: 382/225
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method of automatically classifying images in a consumer digital image collection, includes generating an event representation of the image collection; computing global time-based features for each event within the hierarchical event representation; computing content-based features for each image in an event within the hierarchical event representation; combining content-based features for each image in an event to generate event-level content-based features; and using time-based features and content-based features for each event to classify an event into one of a pre-determined set of semantic categories.

Claims

1. A method comprising:

generating, using a processor, an event representation of an image collection;

computing global time-based features for each event within a hierarchical event representation;

computing image-level content-based features for each image in an event within the hierarchical event representation;

combining image-level content-based features for each image in an event to generate event-level content-based features;

classifying, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories; and

validating, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and

refining the category of at least one event into a sub-category based on the event-level content-based features.

2. The method of claim 1, wherein at least one of the pre-determined set of top-level event categories includes vacation, sports, family moments or social gathering.

3. The method of claim 1, wherein the image-level content-based features include people present, ages of people present, indoor or outdoor, scene type, or materials present.

4. The method of claim 1, wherein the hierarchical event representation includes at least two levels.

5. The method of claim 1, wherein classifying each event comprises using a probabilistic classifier, a rule-based classifier or a combination thereof.

6. The method of claim 1, wherein the validating each event comprises referencing a database of auxiliary factual information associated with subjects identified in the image collection.

7. The method of claim 1, wherein the refining the category of at least one event comprises referencing a database of auxiliary factual information associated with subjects identified in the image collection.

8. The method of claim 1, further comprising calculating a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.

9. The method of claim 8, further comprising determining a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.

10. The method of claim 1, further comprising determining sub-events for each event.

11. A system comprising:

one or more electronic processors configured to;

generate an event representation of an image collection;

compute global time-based features for each event within a hierarchical event representation;

compute image-level content-based features for each image in an event within the hierarchical event representation;

combine image-level content-based features for each image in an event to generate event-level content-based features;

classify, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories;

validate, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and

refine the category of at least one event into a sub-category based on the event-level content-based features.

12. The system of claim 11, wherein the one or more processors are further configured to reference a database of auxiliary factual information associated with subjects identified in the image collection to validate each event.

13. The system of claim 11, wherein the one or more processors are further configured to reference a database of auxiliary factual information associated with subjects identified in the image collection to refine the category of at least one event.

14. The system of claim 11, wherein the one or more processors are further configured to calculate a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.

15. The system of claim 14, wherein the one or more processors are further configured to determine a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.

16. A non-transitory computer-readable medium having instructions stored thereon, the instructions comprising:

instructions to generate an event representation of an image collection;

instructions to compute global time-based features for each event within a hierarchical event representation;

instructions to compute image-level content-based features for each image in an event within the hierarchical event representation;

instructions to combine image-level content-based features for each image in an event to generate event-level content-based features;

instructions to classify, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories;

instructions to validate, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and

instructions to refine the category of at least one event into a sub-category based on the event-level content-based features.

17. The non-transitory computer-readable medium of claim 16, wherein the instructions to validate each event comprises instructions to reference a database of auxiliary factual information associated with subjects identified in the image collection.

18. The non-transitory computer-readable medium of claim 16, wherein the instructions to refine the category of at least one event comprises instructions to reference a database of auxiliary factual information associated with subjects identified in the image collection.

19. The non-transitory computer-readable medium of claim 16, wherein the instructions to classify each event comprises instructions to calculate a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions to classify each event comprises instructions to determine a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.