Classifying and annotating images based on user context

• US 9,189,707 B2
• Filed: 02/24/2014
• Issued: 11/17/2015
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Invent.ly LLC
• Original Assignee: Invent.ly LLC
• Inventors: Brown, Stephen J.
• Primary Examiner(s): Rahmjoo, Mike
• Application Number:
  US14/188,035
• Publication Number:
  US 20150242706A1
• Time in Patent Office: 631 Days
• Field of Search: 382/224
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Annotating and classifying an image based on a user context includes determining a location data of an object captured in an image, determining an attribute data of the object, obtaining sensor data from sensors that are associated with the location data based on the attribute data, determining a recommended user context from one or more predefined user contexts based on a comparison of the location data, the attribute data, and the sensor data with location data, attribute data, and sensor data of one or more images associated with the one or more predefined user contexts, determining a recommended class of the captured image based on the recommended user context, selecting one or more annotation data from the location data, the attribute data, and the sensor data based on the recommended class or the recommended user context, and annotating the image with the one or more annotation data.

Claims

1. A method for annotating and classifying an image based on a user context, said method comprising:

determining, by a location data determining unit, a location data of an object captured in an image by a camera enabled device;

determining, by a processor, an attribute data of said object based on said image;

obtaining sensor data from sensors that are associated with said location data based on said attribute data;

determining, by said processor, a recommended user context from a plurality of predefined user contexts based on a comparison of said location data, said attribute data, and said sensor data with location data, attribute data, and sensor data of a plurality of images associated with each of said plurality of predefined user contexts;

determining, by said processor, a recommended class of said captured image based on said recommended user context;

selecting, by said processor, a plurality of annotation data from said location data of said object, said attribute data of said object, and said sensor data from said sensors based on said recommended class or said recommended user context; and

annotating, by said processor, said image with said plurality of annotation data.

2. The method of claim 1, further comprising generating, by said processor, a list of prioritized predefined user contexts based on said comparison, wherein said recommended user context is a highest priority user context from said list of prioritized predefined user contexts.

3. The method of claim 1, further comprising:

generating, by said processor, a recommendation comprising at least one of said recommended user context and said recommended class; and

processing, by said processor, at least one of a user suggested context and a user suggested class.

4. The method of claim 3, further comprising:

modifying, by said processor, a weight associated with at least one of said location data, said attribute data, and said sensor data associated with said captured image to obtain a modified weight based on which said user suggested context or said user suggested class is prioritized over said recommended user context or said recommended class; and

determining, by said processor, a revised recommended user context or a revised recommended class for a subsequent image captured at a subsequent time based on said modified weight.

5. The method of claim 1, wherein said recommended user context and said recommended class of said captured image are determined based on a set of rules that are stored in a database.

6. The method of claim 1, wherein said location data of said object comprises at least one of a distance of said object from said camera enabled device, a camera inclination angle associated with said camera enabled device, a latitude and longitude data of said object, and a timestamp of said image.

7. The method of claim 1, wherein said sensors that are associated with said location data comprise at least one of an environmental sensor, an object sensor, and a social sensor.

8. The method of claim 7, wherein said sensor data obtained from said environmental sensor comprises at least one of weather data, temperature data, pressure data, humidity data, wind data, precipitation data, and data from at least one of a chemical sensor, an allergen sensor, a noise sensor, and a light sensor.

9. The method of claim 7, wherein said sensor data obtained from said object sensor comprises said attribute data of said object in said image.

10. The method of claim 7, wherein said sensor data obtained from said social sensor comprises at least one of private data to identify a person in said image, data associated with people at said location of said object, and data from a social medium.

11. The method of claim 1, wherein said sensor data comprises reviews, articles, electronic messages, information from the World Wide Web, and news from a location of said object.

12. An annotation data server comprising a processor that:

receives a location data of an object captured in an image;

determines an attribute data of said object;

obtains sensor data from sensors that are associated with said location data based on said attribute data;

determines a recommended user context from a plurality of predefined user contexts based on a comparison of said location data, said attribute data, and said sensor data with location data, attribute data, and sensor data of a plurality of images associated with each of said plurality of predefined user contexts;

selects a plurality of annotation data from said location data of said object, said attribute data of said object, and said sensor data from said sensors based on a recommended class of said image or said recommended user context; and

generates a recommendation comprising said annotation data for annotating said image.

13. The server of claim 12, wherein said recommended user context is determined based on a set of rules that are stored in a database.