System and method for determining image similarity

• US 20020131641A1
• Filed: 03/02/2001
• Published: 09/19/2002
• Est. Priority Date: 01/24/2001
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Zhu, Wei, Sotak, George E., Luo, Jiebo, Mehrotra, Rajiv, Gray, Robert T.
• Granted Patent: US 6,826,316 B2
• Time in Patent Office: Days
• US Class Current: 382/218
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method for determining image similarity. The method includes the steps of automatically providing perceptually significant features of main subject or background of a first image; automatically providing perceptually significant features of main subject or background of a second image; automatically comparing the perceptually significant features of the main subject or the background of the first image to the main subject or the background of the second image; and providing an output in response thereto. In the illustrative implementation, the features are provided by a number of belief levels, where the number of belief levels are preferably greater than two. The perceptually significant features include color, texture and/or shape. In the preferred embodiment, the main subject is indicated by a continuously valued belief map. The belief values of the main subject are determined by segmenting the image into regions of homogenous color and texture, computing at least one structure feature and at least one semantic feature for each region, and computing a belief value for all the pixels in the region using a Bayes net to combine the features. In an illustrative application, the inventive method is implemented in an image retrieval system. In this implementation, the inventive method automatically stores perceptually significant features of the main subject or background of a plurality of first images in a database to facilitate retrieval of a target image in response to an input or query image. Features corresponding to each of the plurality of stored images are automatically sequentially compared to similar features of the query image. Consequently, the present invention provides an automatic system and method for controlling the feature extraction, representation, and feature-based similarity retrieval strategies of a content-based image archival and retrieval system based on an analysis of main subject and background derived from a continuously valued main subject belief map.

Claims

1. An image comparison system comprising:

means for automatically providing perceptually significant features of main subject or background of a first image;

means for automatically providing perceptually significant features of main subject or background of a second image; and

means for automatically comparing the perceptually significant features of the main subject or the background of the first image to the main subject or the background of the second image and providing an output in response thereto.

2. The invention of claim 1 wherein the features are provided by a number of belief levels.

3. The invention of claim 2 wherein the number of belief levels are greater than two.

4. The invention of claim 1 wherein the means for automatically providing perceptually significant features of the main subject or background of the first image includes means for automatically identifying main subject or background of the first image.

5. The invention of claim 4 wherein the means for automatically providing perceptually significant features of the main subject or background of the first image includes means for identifying perceptually significant features of the main subject or the background of the first image.

6. The invention of claim 1 wherein the means for automatically providing perceptually significant features of the main subject or background of the second image includes means for automatically identifying main subject or background of the second image.

7. The invention of claim 6 wherein the means for automatically providing perceptually significant features of the main subject or background of the second image includes means for identifying perceptually significant features of the main subject or the background of the second image.

8. The invention of claim 7 including means for automatically providing perceptually significant features of the main subject or background of a plurality of first images.

9. The invention of claim 8 further including a database for storing the perceptually significant features of the main subject or background of the plurality of first images.

10. The invention of claim 9 wherein the second image is an input image.

11. The invention of claim 10 wherein the means for comparing further includes means for automatically comparing the perceptually significant features of the main subject or background of the input image to the plurality of perceptually significant features of the main subject or background of the plurality of first images.

12. The invention of claim 1 wherein at least one of the perceptually significant features includes color.

13. The invention of claim 1 wherein at least one of the perceptually significant features includes texture.

14. The invention of claim 1 wherein at least one of the perceptually significant features includes shape.

15. The invention of claim 1 wherein the main subject is indicated by a continuously valued belief map.

16. The invention of claim 1 wherein the belief values of the main subject are determined by segmenting the image into regions of homogenous color.

17. The invention of claim 16 wherein the belief values of the main subject are determined by computing at least one structure feature for each region.

18. The invention of claim 16 wherein the belief values of the main subject are determined by computing at least one semantic feature for each region.

19. The invention of claim 16 wherein the belief values of the main subject are determined by computing a belief value for all the pixels in the region using a Bayes net to combine the features.

20. The invention of claim 1 wherein the belief values of the main subject are determined by segmenting the image into regions of homogenous texture.

21. The invention of claim 20 wherein the belief values of the main subject are determined by computing at least one structure feature for each region.

22. The invention of claim 20 wherein the belief values of the main subject are determined by computing at least one semantic feature for each region.

23. The invention of claim 20 wherein the belief values of the main subject are determined by computing a belief value for all the pixels in the region using a Bayes net to combine the features.

25. The invention of claim 24 wherein the features are provided by a number of belief levels.

26. The invention of claim 25 wherein the number of belief levels are greater than two.

27. The invention of claim 24 wherein the step of automatically providing perceptually significant features of the main subject or background of the first image includes the step of automatically identifying main subject or background of the first image.

28. The invention of claim 27 wherein the step of automatically providing perceptually significant features of the main subject or background of the first image includes the step of identifying perceptually significant features of the main subject or the background of the first image.

29. The invention of claim 24 wherein the step of automatically providing perceptually significant features of the main subject or background of the second image includes the step of automatically identifying main subject or background of the second image.

30. The invention of claim 29 wherein the step of automatically providing perceptually significant features of the main subject or background of the second image includes the step of identifying perceptually significant features of the main subject or the background of the second image.

31. The invention of claim 30 including the step of automatically providing perceptually significant features of the main subject or background of a plurality of first images.

32. The invention of claim 31 further including the step of storing the perceptually significant features of the main subject or background of the plurality of first images.

33. The invention of claim 32 wherein the second image is an input image.

34. The invention of claim 33 wherein the step of comparing further includes the step of automatically comparing the perceptually significant features of the main subject or background of the input image to the plurality of perceptually significant features of the main subject or background of the plurality of first images.

35. The invention of claim 24 wherein at least one of the perceptually significant features includes color.

36. The invention of claim 24 wherein at least one of the perceptually significant features includes texture.

37. The invention of claim 24 wherein at least one of the perceptually significant features includes shape.

38. The invention of claim 24 wherein the main subject is indicated by a continuously valued belief map.

39. The invention of claim 24 wherein the belief values of the main subject are determined by segmenting the image into regions of homogenous color.

40. The invention of claim 39 wherein the belief values of the main subject are determined by computing at least one structure feature for each region.

41. The invention of claim 39 wherein the belief values of the main subject are determined by computing at least one semantic feature for each region.

42. The invention of claim 39 wherein the belief values of the main subject are determined by computing a belief value for all the pixels in the region using a Bayes net to combine the features.

43. The invention of claim 24 wherein the belief values of the main subject are determined by segmenting the image into regions of homogenous texture.

44. The invention of claim 43 wherein the belief values of the main subject are determined by computing at least one structure feature for each region.

45. The invention of claim 43 wherein the belief values of the main subject are determined by computing at least one semantic feature for each region.

46. The invention of claim 43 wherein the belief values of the main subject are determined by computing a belief value for all the pixels in the region using a Bayes net to combine the features.

48. The invention of claim 47 wherein the retrieved image has a similar main subject.

49. The invention of claim 47 wherein the retrieved image has a similar background.

50. The invention of claim 47 wherein the retrieved image has a similar main subject, but different background.

51. The invention of claim 47 wherein the retrieved image has a different main subject, but similar background.

52. The invention of claim 47 wherein the retrieved image has a similar main subject, and similar background.

53. The invention of claim 47 wherein the main subject is indicated by a continuously valued belief map, higher belief values indicate higher likelihood of main subject.

54. The invention of claim 47 wherein the belief values of the main subject are determined by using the following steps of:

segmenting the image into regions of homogenous color and texture;

computing at least one structures feature and/or one semantic feature for each region; and

computing a belief value for all the pixels in the region using a Bayes net to combine the features.

55. The invention of claim 47 wherein the similarity is computed by one or more perceptually significant depictive features derived from corresponding subject or background areas.

56. The invention of claim 47 wherein the step (c) of automatically extracting the representation of a main subject and a background in terms of one or more perceptually significant depictive features that dominate a perception of the image includes extraction of perceptually significant color-based representation.

57. The invention of claim 47 wherein the step (c) of automatically extracting the representation of a main subject and a background in terms of one or more perceptually significant depictive features that dominate a perception of the image includes extraction of perceptually significant texture-based representation.

58. The invention of claim 47 including the step (d) of storing image representations in a database by organizing the representations in a set of index structures to facilitate efficient searches of the database to select the desired images.

59. The invention of claim 47 wherein the step (e) of retrieving an image from the database based on its similarity to the main subject and/or background of the query image includes conducting a search of the set of index structures to select images that satisfy aL specified similarity constraint.

60. The invention of claim 47 wherein the step (e) of retrieving an image from the database based on its similarity to the main subject and/or background of the query image includes computing the overall similarity of a query image to a selected image as the weighted sum of the similarity of each of the corresponding main subject and background areas.

24. A image comparison method including the steps of:

automatically providing perceptually significant features of main subject or background of a first image;

automatically providing perceptually significant features of main subject or background of a second image; and

automatically comparing the perceptually significant features of the main subject or the background of the first image to the main subject or the background of the second image and providing an output in response thereto.

47. A method for retrieving an image from an image data base, comprising the steps of:

a) identifying a main subject and a background in a query image;

b) identifying a main subject and a background for each image in the data base;

c) automatically extracting the representation of a main subject and a background using one or more perceptually significant depictive features of the query image;

d) automatically extracting the representation of a main subject and a background using one or more perceptually significant depictive features of each image in the database and

e) retrieving an image from the data base based on its similarity to the main subject and/or background of the query image.