Method for associating semantic information with multiple images in an image database environment

• US 20020188602A1
• Filed: 05/07/2001
• Published: 12/12/2002
• Est. Priority Date: 05/07/2001
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Stubler, Peter O., Mehrotra, Rajiv
• Granted Patent: US 6,804,684 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method of generating captions or semantic labels for an acquired image is based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, where the stored images have preexisting captions or labels associated with them. The method includes the steps of: a) acquiring an image for evaluation with respect to the stored images; b) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment; c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing selected stored images with metadata similar to the acquired image; and d) generating captions or labels for the acquired image from the preexisting captions or labels associated with the selected stored images.

Claims

1. A method of generating one or more captions or semantic labels for an acquired image with limited user interaction, said method based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring an image for evaluation with respect to the stored images;

b) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment;

c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

d) generating one or more captions or labels for the acquired image from one or more preexisting captions or labels associated with the selected stored images.

2. The method as claimed in claim 1 wherein the metadata extracted in step b) describes measurable image features of the acquired image.

3. The method as claimed in claim 2 wherein the metadata is derived from objectively measurable low level features including at least one of color and texture.

4. The method as claimed in claim 2 wherein the metadata is derived from objectively measurable features that pertain to one or more regions of the acquired image including at least one of face, skin, sky, grass and open space.

5. The method as claimed in claim 1 wherein the metadata extracted in step b) includes data accompanying the acquired image that was obtained substantially at the time of capture.

6. The method as claimed in claim 5 wherein the data accompanying the acquired image includes at least one of time of capture, date of capture, location of the image and text or code entered at the time of capture.

7. The method as claimed in claim 1 wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step b) automatically extracts metadata from the acquired image describing measurable image features that pertain to one or more regions of the acquired image;

step c) provides selected stored images having metadata similar to the metadata extracted from the acquired image for corresponding regions; and

step d) generates one or more captions or labels for the regions of the acquired image from the preexisting labels or captions associated with the regions of the selected stored images.

8. The method as claimed in claim 1 wherein step d) comprises providing said one or more of the preexisting captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the acquired image.

9. The method as claimed in claim l wherein step d) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the acquired image.

10. A method of generating one or more captions or semantic labels for groups of acquired images with limited user interaction, said method based upon similarity between the acquired images and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring a plurality of images for evaluation with respect to the stored images;

b) automatically extracting metadata from the acquired images without requiring user interaction in the image database environment;

c) clustering the acquired images into groups based upon similarities in their metadata, thereby resulting in grouped images with similar metadata;

d) selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the grouped images; and

e) generating one or more captions or labels for grouped images from one or more preexisting captions or labels associated with the selected stored images.

11. The method as claimed in claim 10 wherein step e) further includes propagating the generated caption or label through all of the acquired images in the group.

12. The method as claimed in claim 10 wherein the metadata extracted in step b) describes measurable image features of the acquired images.

13. The method as claimed in claim 12 wherein the metadata is derived from objectively measurable low level features including at least one of color and texture.

14. The method as claimed in claim 12 wherein the metadata is derived from objectively measurable features that pertain to one or more regions of the acquired image including at least one of face, skin, sky, grass and open space.

15. The method as claimed in claim 10 wherein the metadata extracted in step b) includes data accompanying the acquired image that was obtained substantially at the time of capture.

16. The method as claimed in claim 15 wherein the data accompanying the acquired images include at least one of time of capture, date of capture, location of the image and text or code entered at the time of capture.

17. The method as claimed in claim 10 wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step b) automatically extracts metadata from the acquired images describing measurable image features that pertain to one or more regions of the acquired images;

step d) provides selected stored images having metadata similar to the metadata extracted from the grouped images for corresponding regions; and

step e) generates one or more captions or labels for the regions of the grouped images from the preexisting labels or captions associated with the regions of the selected stored images.

18. The method as claimed in claim 10 wherein step e) comprises providing said one or more of the preexisting captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the grouped images.

19. The method as claimed in claim 10 wherein step e) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the grouped images.

20. A method of generating one or more captions or semantic labels for an acquired image based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring an image for evaluation with respect to the stored images;

b) processing the acquired image to extract metadata describing measurable features in the acquired image;

c) selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

d) generating one or more captions or labels for the acquired image from one or more preexisting captions or labels associated with the selected stored images.

21. The method as claimed in claim 20 wherein the extracted metadata describes globally measurable features such as color or texture.

22. The method as claimed in claim 20 wherein the extracted metadata describes features measurable in a region of the acquired image such as faces, skin, sky, grass or open space.

23. The method as claimed in claim 22 wherein the captions or labels relate to the region of the acquired image.

27. The computer program product as claimed in claim 26 wherein the metadata extracted in step a) describes measurable image features of the acquired image.

28. The computer program product as claimed in claim 26 wherein the metadata extracted in step a) includes data accompanying the acquired image that was obtained substantially at the time of capture.

29. The computer program product as claimed in claim 26 wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step a) automatically extracts metadata from the acquired image describing measurable image features that pertain to one or more regions of the acquired image;

step b) provides selected stored images having metadata similar to the metadata extracted from the acquired image for corresponding regions; and

step c) generates one or more captions or labels for the regions of the acquired image from the preexisting labels or captions associated with the regions of the selected stored images.

30. The computer program product as claimed in claim 26 wherein step c) comprises providing said one or more of the preexisting captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the acquired image.

31. The computer program product as claimed in claim 26 wherein step c) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the acquired image.

24. A method of assigning one or more captions or semantic labels to an image based upon its similarity with other images, said method comprising the steps of:

a) acquiring a plurality of images;

b) automatically extracting metadata from the images;

c) clustering the images into one or more groups based upon similarities in their metadata, thereby resulting in grouped images having similar underlying features;

d) selecting a respective caption or label appropriate for each group; and

d) automatically extending the respective captions to all the images in each group.

25. A method of assigning one or more captions or labels to acquired images based upon their similarity, said method comprising the steps of:

a) extracting global and regional metadata from the acquired images, said global metadata pertaining to measurable data of the overall images and said regional metadata pertaining to measurable data from one or more regions in the images;

b) clustering the acquired images into groups based upon similarities in their global and regional metadata, thereby resulting in hierarchically grouped images having similar global features at one level and similar regional features at another level in the hierarchy;

c) generating global and regional captions or labels appropriate for at least one of the grouped images; and

d) automatically extending the global captions or labels to other images in the group having the same global metadata and the regional captions or labels to other images in the group having the same regional metadata.

26. A computer program product for generating one or more captions or semantic labels for an acquired image with limited user interaction based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

a) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment;

b) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

c) generating one or more captions or labels for the acquired image from one or more preexisting captions or labels associated with the selected stored images.

32. A computer program product for generating one or more captions or semantic labels for groups of acquired images with limited user interaction based upon similarity between the acquired images and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

a) automatically extracting metadata from the acquired images without requiring user interaction in the image database environment;

b) clustering the acquired images into groups based upon similarities in their metadata, thereby resulting in grouped images with similar metadata;

c) selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the grouped images; and

d) generating one or more captions or labels for grouped images from one or more preexisting captions or labels associated with the selected stored images.

33. The computer program product as claimed in claim 32 wherein step d) further includes propagating the generated caption or label through all of the acquired images in the group.