Context-based image recognition for consumer market research

• US 9,569,692 B2
• Filed: 10/31/2014
• Issued: 02/14/2017
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Inventors: Xie, Wei, Bivins, Michael Allen, Jones, Steve, Deng, Kevin Keqiang, Schiller, Brian, Sullivan, Jonathan, Terrazas, Alejandro, Sheppard, Michael
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Thomas, Mia M
• Application Number:
  US14/530,129
• Publication Number:
  US 20160125265A1
• Time in Patent Office: 837 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/36 Indoor scenes
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

Example methods, apparatus, systems to perform context-based image recognition for consumer market research are disclosed. Disclosed example methods include comparing a first subset of feature points in a first region of interest of a reference image with a first subset of feature points of a first image residing within a first region corresponding spatially to the first region of interest and located at a first location in the first image to determine a first match value. Disclosed example methods also include, in response to determining the first match value satisfies a first threshold, comparing a second subset of feature points in a second region of interest of the reference image with a second subset of the feature points of the first image residing within a second region of the first image corresponding spatially to the second region of interest to determine a second match value.

Claims

1. A method to determine whether a first image depicts a product, the method comprising:

receiving the first image, the first image captured by a mobile device;

comparing, by executing an instruction with a processor, a first subset of feature points in a first region of interest of a reference image with a first subset of feature points of the first image residing within a first region corresponding spatially to the first region of interest and located at a first location in the first image to determine a first match value;

in response to determining the first match value satisfies a first threshold, comparing, by executing an instruction with the processor, a second subset of feature points in a second region of interest of the reference image with a second subset of the feature points of the first image residing within a second region of the first image corresponding spatially to the second region of interest to determine a second match value; and

in response to determining the second match value satisfies a second threshold lower than the first threshold, associating, by executing an instruction with the processor, the first image with product information associated with the reference image.

2. The method as defined in claim 1, wherein the second region of the first image is located at a second location in the first image, the second location being selected relative to the first location based on a spatial relationship between the first region of interest and the second region of interest in the reference image.

3. The method as defined in claim 2, further including, in response to determining the first match value does not satisfy the first threshold:

selecting a third subset of feature points of the first image within a third region of the first image corresponding spatially to the first region of interest and located at a third location in the first image to determine a third match value; and

in response to determining the third match value satisfies the first threshold, comparing, with the processor, the second subset of feature points in the second region of interest of the reference image with a fourth subset of feature points of the first image residing within a fourth region corresponding spatially to the second region of interest and located at a fourth location in the first image to determine a fourth match value, the fourth location being selected relative to the third location based on the spatial relationship between the first region of interest and the second region of interest in the reference image.

4. The method as defined in claim 1, wherein the comparing of the first subset of feature points in the first region of interest of the reference image with the first subset of feature points of the first image includes:

calculating distances between the first subset of feature points in the first region of interest of the reference image and the first subset of feature points of the first image to determine a percentage of the first subset of feature points in the first region of interest of the reference image that match corresponding feature points from the first subset of feature points of the first image; and

calculating the first match value based on the percentage.

5. The method as defined in claim 1, wherein the reference image is associated with a plurality of regions of interest having respective assigned significance values, and further including:

selecting the first region of interest having a highest significance value out of the plurality of regions of interest associated with the reference image; and

setting the first threshold to correspond to a first significance value associated with the first region of interest.

6. The method as defined in claim 1, wherein the reference image is a first reference image associated with the product, the product is associated with a plurality of different reference images, and further including:

in response to determining the second match value satisfies the second threshold;

incrementing a first index value associated with the first reference image;

decrementing index values associated with respective ones of the plurality of reference images other than the first reference image; and

discarding ones of the plurality of reference images associated with index values that do not satisfy a third threshold.

7. The method as defined in claim 1, further including improving accuracy of an image matcher by utilizing different first and second thresholds adjusted to a number of false positive matches between the reference images and captured images depicting products to be identified.

8. The method as defined in claim 1, further including:

in response to determining the second match value satisfies the second threshold, sending the product information associated with the reference image to the mobile device.

9. An apparatus to determine whether a first digital image depicts a product, the apparatus comprising:

a feature point extractor to determine feature points in the first digital image; and

a reference comparator to;

compare a first subset of feature points in a first region of interest of a digital reference image with a first subset of the feature points of the first digital image residing within a first region corresponding spatially to the first region of interest and located at a first location in the first digital image to determine a first match value;

in response to determining the first match value satisfies a first threshold, compare a second subset of feature points in a second region of interest of the digital reference image with a second subset of the feature points of the first digital image residing within a second region of the first digital image corresponding spatially to the second region of interest to determine a second match value; and

in response to determining the second match value satisfies a second threshold lower than the first threshold, associate the first digital image with product information associated with the digital reference image, at least one of the feature point extractor or the reference comparator including hardware.

10. The apparatus as defined in claim 9, wherein the second region of the first digital image is located at a second location in the first image, the second location being selected relative to the first location based on a spatial relationship between the first region of interest and the second region of interest in the reference image.

11. The apparatus as defined in claim 9, wherein to compare the first subset of feature points in the first region of interest of the reference image with the first subset of feature points of the first image, the reference comparator is further to:

calculate distances between the first subset of feature points in the first region of interest of the reference image and the first subset of feature points of the first digital image to determine a percentage of the first subset of feature points in the first region of interest of the reference image that match corresponding feature points from the first subset of feature points of the first image; and

calculate the first match value based on the percentage.

12. The apparatus as defined in claim 9, wherein the reference image is associated with a plurality of regions of interest having respective assigned significance values, and the reference comparator is further to:

select the first region of interest having a highest significance value out of a plurality of regions of interest associated with the reference image; and

set the first threshold to correspond to a first significance value associated with the first region of interest.

13. The apparatus as defined in claim 9, wherein the reference image is a first reference image associated with the product, the product is associated with a plurality of different reference images, and further including:

a reference image manager to, in response to determining the second match value satisfies the second threshold;

increment a first index value associated with the first reference image;

decrement index values associated with respective ones of the plurality of reference images other than the first reference image; and

discard ones of the plurality of reference images associated with index values that do not satisfy a third threshold.

14. The apparatus as defined in claim 9, wherein, in response to determining the first match value does not satisfy the first threshold, the reference comparator is further to:

select a third subset of the feature points of the first digital image within a third region of the first digital image corresponding spatially to the first region of interest and located at a third location in the first digital image to determine a third match value;

in response to determining the third match value satisfies the first threshold, compare the second subset of feature points in the second region of interest of the reference image with a fourth subset of the feature points of the first digital image residing within a fourth region corresponding spatially to the second region of interest and located at a fourth location in the first digital image to determine a fourth match value, the fourth location being selected relative to the third location based on the spatial relationship between the first region of interest and the second region of interest in the reference image.

15. The apparatus as defined in claim 9, wherein the reference comparator is to improve accuracy of an image matcher by utilizing different first and second thresholds adjusted to a number of false positive matches between the reference image and captured images depicting products to be identified.

16. A tangible computer readable storage medium comprising instructions which, when executed, cause a machine to at least;

receive a first image, the first image captured by a mobile device;

compare a first subset of feature points in a first region of interest of a reference image with a first subset of the feature points of the first image residing within a first region corresponding spatially to the first region of interest and located at a first location in the first image to determine a first match value;

in response to determining the first match value satisfies a first threshold, compare a second subset of feature points in a second region of interest of the reference image with a second subset of the feature points of the first image residing within a second region of the first image corresponding spatially to the second region of interest to determine a second match value; and

in response to determining the second match value satisfies a second threshold lower than the first threshold, associate the first image with product information associated with the reference image.

17. The tangible computer readable medium as defined in claim 16, wherein the second region of the first image is located at a second location in the first image, the second location being selected relative to the first location based on a spatial relationship between the first region of interest and the second region of interest in the reference image.

18. The tangible computer readable medium as defined in claim 16, wherein to compare the first subset of feature points in the first region of interest of the reference image with the first subset of feature points of the first image, the instructions, when executed, further cause the machine to:

calculate distances between the first subset of feature points in the first region of interest of the reference image and the first subset of feature points of the first image to determine a percentage of the first subset of feature points in the first region of interest of the reference image that match corresponding feature points from the first subset of feature points of the first image; and

calculate the first match value based on the percentage.

19. The tangible computer readable medium as defined in claim 16, wherein the reference image is associated with a plurality of regions of interest having respective assigned significance values, and the instructions, when executed, further cause the machine to:

select the first region of interest having a highest significance value out of a plurality of regions of interest associated with the reference image; and

set the first threshold to correspond to a first significance value associated with the first region of interest.

20. The tangible computer readable medium as defined in claim 16, wherein the reference image is a first reference image associated with the product, the product is associated with a plurality of different reference images, and the instructions, when executed, further cause the machine to:

increment a first index value associated with the first reference image;

decrement index values associated with respective ones of the plurality of reference images other than the first reference image; and

discard ones of the plurality of reference images associated with index values that do not satisfy a third threshold.

21. The tangible computer readable medium as defined in claim 16, wherein, in response to determining the first match value does not satisfy the first threshold, the instructions, when executed, further cause the machine to:

select a third subset of the feature points of the first image within a third region of the first image corresponding spatially to the first region of interest and located at a third location in the first image to determine a third match value;

in response to determining the third match value satisfies the first threshold, compare the second subset of feature points in the second region of interest of the reference image with a fourth subset of the feature points of the first image residing within a fourth region corresponding spatially to the second region of interest and located at a fourth location in the first image to determine a fourth match value, the fourth location being selected relative to the third location based on the spatial relationship between the first region of interest and the second region of interest in the reference image.

22. The tangible computer readable medium as defined in claim 16, wherein the first region of interest does not overlap the second region of interest.

23. The tangible computer readable storage medium as defined in claim 16, wherein the instructions, when executed, cause the machine further to improve accuracy of an image matcher by utilizing different first and second thresholds adjusted to a number of false positive matches between the reference images and captured image depicting products to be identified.

24. The tangible computer readable medium as defined in claim 16, wherein the instructions, when executed, cause the machine further to:

in response to determining the second match value satisfies the second threshold, sending the product information associated with the reference image to the mobile device.