Automatic and semi-automatic image classification, annotation and tagging through the use of image acquisition parameters and metadata

• US 8,520,909 B2
• Filed: 03/11/2010
• Issued: 08/27/2013
• Est. Priority Date: 03/11/2009
• Status: Active Grant

• Current Assignee: Hong Kong Baptist University
• Original Assignee: Hong Kong Baptist University
• Inventors: Leung, Clement Ho Cheung, Wong, Chun Fan, Liu, Jiming
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US13/255,241
• Publication Number:
  US 20110317885A1
• Time in Patent Office: 1,265 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method for characterizing image contents automatically or semi-automatically using image acquisition parameters and metadata is presented. The method establishes probabilistic and deterministic relationships between different types of metadata and the semantic attributes and contents of images. It furnishes a mechanism that enables the automatic and semi-automatic classification, annotation, tagging, indexing, searching, identification or retrieval of images based on their contents, semantic properties and metadata characteristics. The method uses, but is not limited to, image capture metadata such as focal length, exposure time, relative aperture, flash information, ISO setting, angle of view, subject distance, timestamp, GPS information as well as other forms of metadata, including but not limited to, captions, keywords, headings, tags, comments, remarks, titles which may be automatically, semi-automatically, or manually generated. The present invention can be applied to image databases, web searching, personal search, community search, broad-based or vertical search engines for internet, intranet, extranet or other usages.

Claims

1. A method of semantically characterizing the semantic attributes and contents of digital or non-digital images by

using characterization rules to determine image contents and properties from image acquisition metadata, data affiliated with the image, and/or the image itself, the image, image acquisition metadata and data affiliated with the image being stored on a non-transitory computer-readable storage medium, and

using probabilistic or deterministic relationships among the metadata and/or affiliated data to create a semantic image characterization which is used to enrich pre-existing metadata and/or affiliated data or is used to index the image contents in a computer readable storage medium in order to facilitate future retrieval of the image using semantic terms via a data processing apparatus,

wherein the image acquisition metadata is selected from at least one of focal length, exposure time, relative aperture, flash information, ISO setting, angle of view, subject distance, timestamp, and GPS information, and

wherein the characterization rules comprise one or more rules which infer image contents and properties by determining that when focal length belongs to a specific set of values, and exposure time belongs to a specific set of values, and subject distance belongs to a specific set of values, and timestamp belongs to a specific set of values, and relative aperture belongs to a specific set of values, that the image is of a certain type of scene or contains certain types of contents or having certain properties.

2. A method according to claim 1 wherein the affiliated data is selected from at least one of contents classification, tags, annotations, captions, keywords, headings, comments, remarks, titles, related texts, surrounding texts, or linked text.

3. The method of claim 1 wherein the characterization may be enhanced by extracting and processing the affiliated data.

4. The method of claim 1 wherein the images may be Web images, non-Web images, images located in other public or private image repositories, and the method can be applied to image databases, web searching, personal search, community search, broad-based or vertical search engines for internet, intranet, extranet or other usages.

5. The method of claim 1 wherein the characterization may be enhanced by correlating metadata and/or affiliated data with external or internal databases.

6. The method of claim 5 wherein the correlation comprises using GPS coordinates and timestamp metadata to determine the weather or news information from a weather or news database for the place and time of the image, or from other databases or geographic information systems.

7. The method of claim 1 wherein the semantic characterization includes one or more of objects in the image, relationships among objects in the image, attributes or properties of objects or relationships in the image, scene in the image, environment in the image, context of the image, landmarks in the image, location where the image is taken, time when the image is taken, background in the image, features in the image, occasions in the image, events in the image, reasons why the image was taken, living things and non-living things in the image, mood of people in the image, or actions in the image.

8. The method of claim 1 wherein the image metadata and/or affiliated data may be automatically, semi-automatically or manually generated.

9. The method of claim 1 wherein the characterization may be enhanced by applying known image processing and/or face recognition algorithms.

10. The method according to claim 1 wherein the characterization rules makes use of conjunction and/or disjunction in combining properties of the image acquisition metadata.

11. The method of claim 1 wherein the image acquisition metadata comprises EXIF (Exchangeable Image File Format) metadata.

12. The method of claim 1 wherein characterization of the image results from analysis of the image comprises the detection of faces, the recognition of faces, the recognition of fingerprints or recognition of other biometric data.

13. The method of claim 12 wherein annotations may be added to the image from databases of biometric data.

14. The method of claim 13 wherein the biometric data comprising facial features or fingerprints.

15. The method of claim 1 wherein characterization of the image comprising analysis of the image using image processing algorithms to determine shape, color, feature, or texture of the image.

16. The method of claim 15 wherein the image processing algorithm comprising SIFT (scale-invariant feature transformation).

17. The method of claim 1 wherein enriching pre-existing metadata comprises automatically populating one or more fields within image processing and classification standards such as the MPEG-7 standard.

18. The method of claim 17 wherein the one or more fields are selected from datatypes comprising Structured Annotation Datatype, Keyword Annotation Datatype, and Text Annotation Datatype.