Automatic and Semi-automatic Image Classification, Annotation and Tagging Through the Use of Image Acquisition Parameters and Metadata

• US 20110317885A1
• Filed: 03/11/2010
• Published: 12/29/2011
• Est. Priority Date: 03/11/2009
• Status: Active Grant

• Current Assignee: Hong Kong Baptist University
• Original Assignee: Hong Kong Baptist University
• Inventors: Leung, Clement Ho Cheung, Liu, Jiming, Wong, Chun Fan
• Granted Patent: US 8,520,909 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method for characterizing image contents automatically or semi-automatically using image acquisition parameters and metadata is presented. The method establishes probabilistic and deterministic relationships between different types of metadata and the semantic attributes and contents of images. It furnishes a mechanism that enables the automatic and semi-automatic classification, annotation, tagging, indexing, searching, identification or retrieval of images based on their contents, semantic properties and metadata characteristics. The method uses, but is not limited to, image capture metadata such as focal length, exposure time, relative aperture, flash information, ISO setting, angle of view, subject distance, timestamp, GPS information as well as other forms of metadata, including but not limited to, captions, keywords, headings, tags, comments, remarks, titles which may be automatically, semi-automatically, or manually generated. The present invention can be applied to image databases, web searching, personal search, community search, broad-based or vertical search engines for internet, intranet, extranet or other usages.

Claims

1. A method of semantically characterizing the semantic attributes and contents of digital or non-digital images by analyzing image acquisition metadata, data affiliated with the image, and/or the image itself, the image, image acquisition metadata and data affiliated with the image being stored on a non-transitory computer-readable storage medium, and using probabilistic or deterministic relationships among the metadata and/or affiliated data to create a semantic image characterization which is used to enrich pre-existing metadata and/or affiliated data or is used to index the image contents in a computer readable storage medium in order to facilitate future retrieval of the image using semantic terms via a data processing apparatus.

2. A method according to claim 1 wherein the image acquisition metadata is selected from at least one of focal length, exposure time, relative aperture, flash information, ISO setting, angle of view, subject distance, timestamp, or GPS information.

3. A method according to claim 1 wherein the affiliated data is selected from at least one of contents classification, annotation, tags, annotations, captions, keywords, headings, comments, remarks, titles, related texts, surrounding texts, or linked text.

4. The method of claim 2 wherein the characterization uses rules, and/or other computational procedures and algorithms.

5. The method of claim 1 wherein the images may be Web images, non-Web images, images located in other public or private image repositories, and the method can be applied to image databases, web searching, personal search, community search, broad-based or vertical search engines for internet, intranet, extranet or other usages.

6. The method of claim 2 wherein the characterization may be enhanced by correlating metadata and/or affiliated data with external or internal databases.

7. The method of claim 6 wherein the correlation comprises using GPS coordinates and timestamp metadata to determine the weather or news information from a weather or news database for the place and time of the image, or from other geographic information systems or databases.

8. The method of claim 1 wherein the semantic characterization includes one or more of objects in the image, relationships among objects in the image, attributes or properties of objects or relationships in the image, scene in the image, environment in the image, context of the image, landmarks in the image, location where the image is taken, time when the image is taken, background in the image, features in the image, occasions in the image, events in the image, reasons why the image was taken, living things and non-living things in the image, mood of people in the image, or actions in the image.

9. The method of claim 1 wherein the image metadata and/or affiliated data may be automatically, semi-automatically or manually generated.

10. The method of claim 4 wherein the characterization rule infers image content from image acquisition metadata by determining that when focal length belongs to a specific set of values, and exposure time belongs to a specific set of values, and subject distance belongs to a specific set of values, and timestamp belongs to a specific set of values, and relative aperture belongs to a specific set of values, that the image is of a certain type of scene or contains certain types of contents.

11. The method according to claim 10 wherein the characterization rule makes use of conjunction and/or disjunction in combining properties of the image acquisition metadata.

12. The method of claim 1 wherein the image acquisition metadata includes EXIF (Exchangeable Image File Format) metadata.

13. The method of claim 1 wherein characterization of the image results from analysis of the image including the detection or faces, the recognition of faces, the recognition of fingerprints or recognition of other biometric data.

14. The method of claim 13 wherein annotations may be added to the image from databases of biometric data.

15. The method of claim 14 wherein the biometric data is facial features or fingerprints.

16. The method of claim 1 wherein characterization of the image includes analysis of the image using image processing algorithms to determine shape, color, feature, or texture of the image.

17. The method of claim 14 wherein the image processing algorithm is SIFT (scale-invariant feature transformation).

18. The method of claim 1 wherein enriching pre-existing metadata comprises automatically populating one or more fields within the MPEG-7 standard.

19. The method of claim 16 wherein the MPEG-7 fields are selected from Structured Annotation Datatype, Keyword Annotation Datatype, and/or Text Annotation Datatype.