Method for associating semantic information with multiple images in an image database environment

• US 6,804,684 B2
• Filed: 05/07/2001
• Issued: 10/12/2004
• Est. Priority Date: 05/07/2001
• Status: Expired due to Term

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Stubler, Peter O., Mehrotra, Rajiv
• Primary Examiner(s): Robinson, Greta
• Assistant Examiner(s): Le, Miranda
• Application Number:
  US09/850,475
• Publication Number:
  US 20020188602A1
• Time in Patent Office: 1,254 Days
• Field of Search: 707/3, 707/5, 707/102, 707/104.1, 382/180-182, 382/190, 382/206, 382/219, 382/175, 715/512
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method of generating captions or semantic labels for an acquired image is based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, where the stored images have preexisting captions or labels associated with them. The method includes the steps of: a) acquiring an image for evaluation with respect to the stored images; b) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment; c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing selected stored images with metadata similar to the acquired image; and d) generating captions or labels for the acquired image from the preexisting captions or labels associated with the selected stored images.

Claims

1. A method of generating one or more captions or semantic labels for an acquired image with limited user interaction, said method based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring an image for evaluation with respect to the stored images;

b) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment;

c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

d) automatically generating one or more captions or labels for the acquired image by automatically, without any user intervention, extending the captions or labels to the acquired image from one or more preexisting captions or labels associated with the selected stored images;

wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step b) automatically extracts metadata from the acquired image describing measurable image features that pertain to one or more regions of the acquired image;

step c) provides selected stored images having metadata similar to the metadata extracted from the acquired image for corresponding regions; and

step d) generates one or more captions or labels for the regions of the acquired image from the preexisting labels or captions associated with the regions of the selected stored images.

2. The method as claimed in claim 1 wherein the metadata is derived from objectively measurable low level features including at least one of color and texture.

3. The method as claimed in claim 1 wherein the metadata is derived from objectively measurable features that pertain to one or more regions of the acquired image including at least one of face, skin, sky, grass and open space.

4. The method as claimed in claim 1 wherein the metadata extracted in step b) includes data accompanying the acquired image that was obtained substantially at the time of capture.

5. The method as claimed in claim 4 wherein the data accompanying the acquired image includes at least one of time of capture, date of capture, location of the image and text or code entered at the time of capture.

6. The method as claimed in claim 1 wherein the metadata extracted in step b) describes measurable image features of the acquired image.

7. The method as claimed in claim 1 wherein step d) comprises providing said one or more of the automatically-extended captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the acquired image.

8. The method as claimed in claim 1 wherein step d) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the acquired image.

9. A method of generating one or more captions or semantic labels for groups of acquired images with limited user interaction, said method based upon similarity between the acquired images and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring a plurality of images for evaluation with respect to the stored images;

b) automatically extracting metadata from the acquired images without requiring user interaction in the image database environment;

c) automatically clustering the acquired images into groups based upon similarities in their metadata, thereby resulting in grouped images with similar metadata;

d) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the grouped images; and

e) automatically generating one or more captions or labels for grouped images by automatically, without any user intervention, extending the captions or labels to the grouped images from one or more preexisting captions or labels associated with the selected stored images;

wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step b) automatically extracts metadata from the acquired images describing measurable image features that pertain to one or more regions of the acquired images;

step d) provides selected stored images having metadata similar to the metadata extracted from the grouped images for corresponding regions; and

step e) generates one or more captions or labels for the regions of the grouped images from the preexisting labels or captions associated with the regions of the selected stored images.

10. The method as claimed in claim 9 wherein the metadata extracted in step b) describes measurable image features of the acquired images.

11. The method as claimed in claim 10 wherein the metadata is derived from objectively measurable low level features including at least one of color and texture.

12. The method as claimed in claim 10 wherein the metadata is derived from objectively measurable features that pertain to one or more regions of the acquired image including at least one of face, skin, sky, grass and open space.

13. The method as claimed in claim 9 wherein the metadata extracted in step b) includes data accompanying the acquired image that was obtained substantially at the time of capture.

14. The method as claimed in claim 13 wherein the data accompanying the acquired images include at least one of time of capture, date of capture, location of the image and text or code entered at the time of capture.

15. The method as claimed in claim 9 wherein step e) further includes propagating the generated caption or label through all of the acquired images in the group.

16. The method as claimed in claim 9 wherein step e) comprises providing said one or more of the automatically-extended captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the grouped images.

17. The method as claimed in claim 9 wherein step e) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the grouped images.

18. A method of generating one or more captions or semantic labels for an acquired image based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said method comprising the steps of:

a) acquiring an image for evaluation with respect to the stored images;

b) processing the acquired image to extract metadata describing measurable features in the acquired image;

c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

d) automatically generating one or more captions or labels for the acquired image by automatically, without any user intervention, extending the captions or labels to the acquired image from one or more preexisting captions or labels associated with the selected stored images;

wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step b) automatically extracts metadata from the acquired image describing measurable image features that pertain to one or more regions of the acquired image;

step c) provides selected stored images having metadata similar to the metadata extracted from the acquired image for corresponding regions; and

step d) generates one or more captions or labels for the regions of the acquired image from the preexisting labels or captions associated with the regions of the selected stored images.

19. The method as claimed in claim 18 wherein the extracted metadata describes globally measurable features such as color or texture.

20. The method as claimed in claim 18 wherein the extracted metadata describes features measurable in a region of the acquired image such as faces, skin, sky, grass or open space.

21. The method as claimed in claim 20 wherein the captions or labels relate to the region of the acquired image.

22. A computer program product for generating one or more captions or semantic labels for an acquired image with limited user interaction based upon similarity between the acquired image and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

a) automatically extracting metadata from the acquired image without requiring user interaction in the image database environment;

b) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the acquired image; and

c) automatically generating one or more captions or labels for the acquired image by automatically, without any user intervention, extending the captions or labels to the acquired image from one or more preexisting captions or labels associated with the selected stored images;

wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step a) automatically extracts metadata from the acquired image describing measurable image features that pertain to one or more regions of the acquired image;

step b) provides selected stored images having metadata similar to the metadata extracted from the acquired image for corresponding regions; and

step c) generates one or more captions or labels for the regions of the acquired image from the preexisting labels or captions associated with the regions of the selected stored images.

23. The computer program product as claimed in claim 22 wherein the metadata extracted in step a) includes data accompanying the acquired image that was obtained substantially at the time of capture.

24. The computer program product as claimed in claim 22 wherein the metadata extracted in step a) describes measurable image features of the acquired image.

25. The computer program product as claimed in claim 22 wherein step c) comprises providing said one or more of the automatically-extended captions or labels associated with the selected stored images for user verification or correction before using them as captions or labels for the acquired image.

26. The computer program product as claimed in claim 22 wherein step c) comprises extending one or more of the preexisting captions or labels associated with the selected stored images as the caption or label for the acquired image.

27. A computer program product for generating one or more captions or semantic labels for groups of acquired images with limited user interaction based upon similarity between the acquired images and one or more stored images that are maintained in an image database environment, wherein the stored images have preexisting captions or labels associated with them, said computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

a) automatically extracting metadata from the acquired images without requiring user interaction in the image database environment;

b) clustering the acquired images into groups based upon similarities in their metadata, thereby resulting in grouped images with similar metadata;

c) automatically selecting one or more stored images having metadata similar to the extracted metadata, thereby providing one or more selected stored images with metadata similar to the grouped images; and

d) automatically generating one or more captions or labels for grouped images by automatically, without any user intervention, extending the captions or labels to the grouped images from one or more preexisting captions or labels associated with the selected stored images;

wherein the stored images have one or more preexisting captions or labels associated with one or more regions of the stored images and wherein step a) automatically extracts metadata from the acquired images describing measurable image features that pertain to one or more regions of the acquired images;

step c) provides selected stored images having metadata similar to the metadata extracted from the grouped images for corresponding regions; and

step d) generates one or more captions or labels for the regions of the grouped images from the preexisting labels or captions associated with the regions of the selected stored images.

28. The computer program product as claimed in claim 27 wherein step d) further includes propagating the generated caption or label through all of the acquired images in the group.