Automated techniques for comparing contents of images

• US 7,236,632 B2
• Filed: 04/11/2003
• Issued: 06/26/2007
• Est. Priority Date: 04/11/2003
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/412,757
• Publication Number:
  US 20040202349A1
• Time in Patent Office: 1,537 Days
• Field of Search: 382/209, 382217-220, 382224-227, 382/162, 382/108, 382/168, 382/170, 382/305, 382/181, 382/190, 382/173, 707 1- 6, 707/104.1, 358403-404
• US Class Current: 382/218
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Automated techniques for comparing contents of images. For a given image (referred to as an “input image”), a set of images (referred to as “a set of candidate images”) are processed to determine if the set of candidate images comprises an image whose contents or portions thereof match contents included in a region of interest in the input image.

Claims

1. A computer-implemented method of determining if a first set of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the method comprising:

determining if the input image and images in the first set of images have a first characteristic;

determining if the input image and the images in the first set of images have a second characteristic;

identifying a first set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and images in the first set of images have the first characteristic and the second characteristic;

comparing contents of the input image and contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques; and

determining, based upon results of the comparison, if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

2. The method of claim 1 wherein the region of interest corresponds to a portion of the input image.

3. The method of claim 1 wherein:

determining if the input image and the images in the first set of images have the first characteristic comprises determining if the input image and the images in the first set of images are segmented; and

determining if the input image and the images in the first set of images have the second characteristic comprises determining if the input image and the images in the first set of images comprise text that can be recognized.

4. The method of claim 3 wherein determining if the input image and the images in the first set of images are segmented comprises receiving information indicating whether the input image and images in the set of candidate images are segmented.

5. The method of claim 3 wherein determining if the input image and the images in the first set of images are segmented comprises:

identifying one or more capture devices used to capture the input image and the images in the first set of images; and

determining, based upon the one or more capture devices, whether the input image and images in the first set of images are segmented.

6. The method of claim 5 further comprising:

storing information identifying a plurality of capture devices; and

storing, for each capture device in the plurality of capture devices, information indicating if an image captured by the capture device is segmented.

7. The method of claim 3 wherein determining if the input image and the images in the first set of images are segmented comprises:

processing the input image and the images in the first set of images to determine whether the input image and the images in the first set of images are segmented.

8. The method of claim 3 wherein determining if the input image and the images in the first set of images comprise text that can be recognized comprises receiving information indicative of whether the input image and the images in the first set of images comprise text that can be recognized.

9. The method of claim 3 wherein determining if the input image and the images in the first set of images comprise text that can be recognized comprises:

identifying one or more capture devices used to capture the input image and the images in the first set of images; and

determining, based upon the one or more capture devices, whether the input image and the images in the first set of images comprise text that can be recognized.

10. The method of claim 3 wherein determining if the input image and the images in the first set of images comprise text that can be recognized comprises processing the input image and the images in the first set of images to determine whether the input image and the images in the set of candidate images comprise text that can be recognized.

11. The method of claim 3 further comprising:

storing first information identifying one or more comparison techniques from the plurality of comparison techniques to be selected based upon whether or not the input image and the images in the first set of images have the first characteristic and the second characteristic.

12. The method of claim 11 wherein identifying the first set of comparison techniques from the plurality of comparison techniques comprises:

using the first information to determine if a comparison technique from the plurality of comparison techniques is included in the first set of comparison techniques.

13. The method of claim 3 further comprising:

receiving information identifying a time constraint; and

identifying a second set of comparison techniques from the first set of comparison techniques that satisfy the time constraint;

wherein comparing the contents of the input image and the contents of the images in the first set of images using the one or more comparison techniques from the first set of comparison techniques comprises using comparison techniques included in the second set of comparison techniques.

14. The method of claim 3 wherein comparing the contents of the input image and the contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques comprises:

determining an order in which comparison techniques in the first set of comparison techniques are to be applied; and

applying the one or more comparison techniques from the first set of comparison techniques in the determined order.

15. The method of claim 3 wherein determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image; and

identifying, based upon the confidence scores for the one or more comparison techniques, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

16. The method of claim 15 wherein the image from the matching images identified as comprising contents that match the presentation information in the region of interest of the input image is a matching image identified by a comparison technique with the highest confidence score.

17. The method of claim 3 wherein determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques,

a confidence score for the comparison technique,

a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image, and

a distance value for each image in the first set of images, the distance value indicative of a measure of closeness of match between the presentation information contained in the region of interest of the input image and contents in the image from the first set of images; and

identifying, based upon the confidence scores for the one or more comparison techniques and the distance values, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

18. A computer-implemented method of determining if a plurality of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the plurality of images including images captured by one or more types of capture devices, the method comprising:

determining, for each image in the plurality of images, a type of capture device used to capture the image;

grouping the images in the plurality of images into a plurality of sets based upon the types of capture devices used to capture the images such that all images captured by a type of capture device are grouped in one set;

for each set in the plurality of sets of images,

determining if the input image and images in the set have a first characteristic,

determining if the input image and the images in the set have a second characteristic,

identifying a set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and the images in the set have the first characteristic and the second characteristic,

comparing contents of the input image and contents of the images in the set using one or more comparison techniques from the set of comparison techniques,

receiving, for each comparison technique in the one or more comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the set of images as containing contents that match the presentation information in the region of interest of the input image,

determining a set confidence score for the set based upon the confidence scores for the one or more comparison techniques, and

determining, from the matching images identified by the one or more comparison techniques, a matching image identified by the comparison technique with the highest confidence score as a matching image for the set; and

determining, based upon the set confidence scores for the plurality of sets, if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

19. The method of claim 18 wherein determining if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

identifying the matching image for the set with the highest set confidence score as an image comprising contents that match the presentation information in the region of interest of the input image.

20. The method of claim 18 wherein:

determining if the input image and images in the set have the first characteristic comprises determining if the input image and the images in the set are segmented; and

determining if the input image and the images in the set have a second characteristic comprises determining if the input image and the images in the set comprise text that can be recognized.

21. A data processing system for determining if a first set of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the data processing system comprising:

a processor;

a memory coupled to the processor, the memory configured to store a plurality of code modules for execution by the processor, the plurality of code modules comprising;

a code module for determining if the input image and images in the first set of images have a first characteristic;

a code module for determining if the input image and the images in the first set of images have a second characteristic;

a code module for identifying a first set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and images in the first set of images have the first characteristic and the second characteristic;

a code module for comparing contents of the input image and contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques; and

a code module for determining, based upon results of the comparison, if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

22. The system of claim 21 wherein the region of interest corresponds to a portion of the input image.

23. The system of claim 21 wherein:

the code module for determining if the input image and the images in the first set of images have the first characteristic comprises a code module for determining if the input image and the images in the first set of images are segmented; and

the code module for determining if the input image and the images in the first set of images have the second characteristic comprises a code module for determining if the input image and the images in the first set of images comprise text that can be recognized.

24. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images are segmented comprises a code module for receiving information indicating whether the input image and images in the set of candidate images are segmented.

25. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images are segmented comprises:

a code module for identifying one or more capture devices used to capture the input image and the images in the first set of images; and

a code module for determining, based upon the one or more capture devices, whether the input image and images in the first set of images are segmented.

26. The system of claim 25 wherein the memory is configured to store information identifying a plurality of capture devices, and for each capture device in the plurality of capture devices, information indicating if an image captured by the capture device is segmented.

27. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images are segmented comprises:

a code module for processing the input image and the images in the first set of images to determine whether the input image and the images in the first set of images are segmented.

28. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images comprise text that can be recognized comprises a code module for receiving information indicative of whether the input image and the images in the first set of images comprise text that can be recognized.

29. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images comprise text that can be recognized comprises:

a code module for identifying one or more capture devices used to capture the input image and the images in the first set of images; and

a code module for determining, based upon the one or more capture devices, whether the input image and the images in the first set of images comprise text that can be recognized.

30. The system of claim 23 wherein the code module for determining if the input image and the images in the first set of images comprise text that can be recognized comprises a code module for processing the input image and the images in the first set of images to determine whether the input image and the images in the set of candidate images comprise text that can be recognized.

31. The system of claim 23 wherein the memory is configured to store first information identifying one or more comparison techniques from the plurality of comparison techniques to be selected based upon whether or not the input image and the images in the first set of images have the first characteristic and the second characteristic.

32. The system of claim 31 wherein the code module for identifying the first set of comparison techniques from the plurality of comparison techniques comprises:

a code module for using the first information to determine if a comparison technique from the plurality of comparison techniques is included in the first set of comparison techniques.

33. The system of claim 23 wherein the plurality of code modules further comprises:

a code module for receiving information identifying a time constraint; and

a code module for identifying a second set of comparison techniques from the first set of comparison techniques that satisfy the time constraint;

wherein the code module for comparing the contents of the input image and the contents of the images in the first set of images using the one or more comparison techniques from the first set of comparison techniques comprises a code module for using comparison techniques included in the second set of comparison techniques.

34. The system of claim 23 wherein the code module for comparing the contents of the input image and the contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques comprises:

a code module for determining an order in which comparison techniques in the first set of comparison techniques are to be applied; and

a code module for applying the one or more comparison techniques from the first set of comparison techniques in the determined order.

35. The system of claim 23 wherein the code module for determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

a code module for receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image; and

a code module for identifying, based upon the confidence scores for the one or more comparison techniques, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

36. The system of claim 35 wherein the image from the matching images identified as comprising contents that match the presentation information in the region of interest of the input image is a matching image identified by a comparison technique with the highest confidence score.

37. The system of claim 23 wherein the code module for determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

a code module for receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques,

a confidence score for the comparison technique,

a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image, and

a distance value for each image in the first set of images, the distance value indicative of a measure of closeness of match between the presentation information contained in the region of interest of the input image and contents in the image from the first set of images; and

a code module for identifying, based upon the confidence scores for the one or more comparison techniques and the distance values, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

38. A computer system for determining if a plurality of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the plurality of images including images captured by one or more types of capture devices, the computer system comprising:

a processor;

a memory coupled to the processor, the memory configured to store a plurality of code modules for execution by the processor, the plurality of code modules comprising;

a code module for determining, for each image in the plurality of images, a type of capture device used to capture the image;

a code module for grouping the images in the plurality of images into a plurality of sets based upon the types of capture devices used to capture the images such that all images captured by a type of capture device are grouped in one set;

for each set in the plurality of sets of images,

a code module for determining if the input image and images in the set have a first characteristic,

a code module for determining if the input image and the images in the set have a second characteristic,

a code module for identifying a set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and the images in the set have the first characteristic and the second characteristic,

a code module for comparing contents of the input image and contents of the images in the set using one or more comparison techniques from the set of comparison techniques,

a code module for receiving, for each comparison technique in the one or more comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the set of images as containing contents that match the presentation information in the region of interest of the input image,

a code module for determining a set confidence score for the set based upon the confidence scores for the one or more comparison techniques, and

a code module for determining, from the matching images identified by the one or more comparison techniques, a matching image identified by the comparison technique with the highest confidence score as a matching image for the set; and

a code module for determining, based upon the set confidence scores for the plurality of sets, if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

39. The system of claim 38 wherein the code module for determining if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

a code module for identifying the matching image for the set with the highest set confidence score as an image comprising contents that match the presentation information in the region of interest of the input image.

40. The system of claim 38 wherein:

the code module for determining if the input image and images in the set have the first characteristic comprises a code module for determining if the input image and the images in the set are segmented; and

the code module for determining if the input image and the images in the set have a second characteristic comprises a code module for determining if the input image and the images in the set comprise text that can be recognized.

41. A computer program product stored on a computer-readable medium for determining if a first set of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the computer program product comprising:

code for determining if the input image and images in the first set of images have a first characteristic;

code for determining if the input image and the images in the first set of images have a second characteristic;

code for identifying a first set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and images in the first set of images have the first characteristic and the second characteristic;

code for comparing contents of the input image and contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques; and

code for determining, based upon results of the comparison, if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

42. The computer program product of claim 41 wherein the region of interest corresponds to a portion of the input image.

43. The computer program product of claim 41 wherein:

the code for determining if the input image and the images in the first set of images have the first characteristic comprises code for determining if the input image and the images in the first set of images are segmented; and

the code for determining if the input image and the images in the first set of images have the second characteristic comprises code for determining if the input image and the images in the first set of images comprise text that can be recognized.

44. The computer program product of claim 43 further comprising:

code for storing first information identifying one or more comparison techniques from the plurality of comparison techniques to be selected based upon whether or not the input image and the images in the first set of images have the first characteristic and the second characteristic.

45. The computer program product of claim 44 wherein the code for identifying the first set of comparison techniques from the plurality of comparison techniques comprises:

code for using the first information to determine if a comparison technique from the plurality of comparison techniques is included in the first set of comparison techniques.

46. The computer program product of claim 43 further comprising:

code for receiving information identifying a time constraint; and

code for identifying a second set of comparison techniques from the first set of comparison techniques that satisfy the time constraint;

wherein the code for comparing the contents of the input image and the contents of the images in the first set of images using the one or more comparison techniques from the first set of comparison techniques comprises code for using comparison techniques included in the second set of comparison techniques.

47. The computer program product of claim 43 wherein the code for comparing the contents of the input image and the contents of images in the first set of images using one or more comparison techniques from the first set of comparison techniques comprises:

code for determining an order in which comparison techniques in the first set of comparison techniques are to be applied; and

code for applying the one or more comparison techniques from the first set of comparison techniques in the determined order.

48. The computer program product of claim 43 wherein the code for determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

code for receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image; and

code for identifying, based upon the confidence scores for the one or more comparison techniques, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

49. The computer program product of claim 48 wherein the image from the matching images identified as comprising contents that match the presentation information in the region of interest of the input image is a matching image identified by a comparison technique with the highest confidence score.

50. The computer program product of claim 43 wherein the code for determining if the first set of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

code for receiving, for each comparison technique in the one or more techniques from the first set of comparison techniques,

a confidence score for the comparison technique,

a matching image identified by the comparison technique from the first set of images as containing contents that match the presentation information in the region of interest of the input image, and

a distance value for each image in the first set of images, the distance value indicative of a measure of closeness of match between the presentation information contained in the region of interest of the input image and contents in the image from the first set of images; and

code for identifying, based upon the confidence scores for the one or more comparison techniques and the distance values, an image, from the matching images identified by the one or more comparison techniques, as the image comprising contents that match the presentation information in the region of interest of the input image.

51. A computer program product stored on a computer-readable medium for determining if a plurality of images includes an image containing contents that match presentation information contained in a region of interest of an input image, the plurality of images including images captured by one or more types of capture devices, the computer program product comprising:

code for determining, for each image in the plurality of images, a type of capture device used to capture the image;

code for grouping the images in the plurality of images into a plurality of sets based upon the types of capture devices used to capture the images such that all images captured by a type of capture device are grouped in one set;

for each set in the plurality of sets of images,

code for determining if the input image and images in the set have a first characteristic,

code for determining if the input image and the images in the set have a second characteristic,

code for identifying a set of comparison techniques from a plurality of comparison techniques based upon whether or not the input image and the images in the set have the first characteristic and the second characteristic,

code for comparing contents of the input image and contents of the images in the set using one or more comparison techniques from the set of comparison techniques,

code for receiving, for each comparison technique in the one or more comparison techniques, a confidence score for the comparison technique and a matching image identified by the comparison technique from the set of images as containing contents that match the presentation information in the region of interest of the input image,

code for determining a set confidence score for the set based upon the confidence scores for the one or more comparison techniques, and

code for determining, from the matching images identified by the one or more comparison techniques, a matching image identified by the comparison technique with the highest confidence score as a matching image for the set; and

code for determining, based upon the set confidence scores for the plurality of sets, if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image.

52. The computer program product of claim 51 wherein the code for determining if the plurality of images includes an image comprising contents that match the presentation information in the region of interest of the input image comprises:

code for identifying the matching image for the set with the highest set confidence score as an image comprising contents that match the presentation information in the region of interest of the input image.

53. The computer program product of claim 51 wherein:

the code for determining if the input image and images in the set have the first characteristic comprises code for determining if the input image and the images in the set are segmented; and

the code for determining if the input image and the images in the set have a second characteristic comprises code for determining if the input image and the images in the set comprise text that can be recognized.