Image recognition verification

• US 9,864,758 B2
• Filed: 05/12/2014
• Issued: 01/09/2018
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Jaber, Mustafa, Song, Bing, Sudol, Jeremi
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US14/275,067
• Publication Number:
  US 20150169638A1
• Time in Patent Office: 1,338 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

Systems and methods of verifying the results of an initial image recognition process are presented. A verification engine can receive a set of candidate images corresponding to the results of an image recognition process performed on a captured query image. The verification engine can determine an appropriate verification technique to apply to the images of the candidate set, and classify, re-rank or otherwise re-organize the candidate set such that the best match from the candidate set is confirmed as a proper match.

Claims

1. A system for verifying a recognized object, comprising:

a candidate database storing a plurality of candidate images;

a verification engine communicatively coupled to the candidate database, configured to;

receive a plurality of candidate results, wherein the plurality of candidate results comprises at least two of the plurality of candidate images and each of the plurality of candidate results corresponds to a potential match for a captured image;

select a verification technique from two or more verification techniques based on one or more image processing techniques or matching methods selected based on one or more characteristics of at least one candidate result from the plurality of candidate results, the one or more characteristics comprising at least one of image design content, image text content, image size, image color, image resolution, image quality, image data type, and image frame rate;

generate, using the selected verification technique, at least one match score for the at least one candidate result as a function of the captured image and the at least one candidate result; and

classify the at least one candidate result based on the at least one generated match score.

2. The system of claim 1, wherein the verification technique comprises at least one image processing technique and at least one matching method.

3. The system of claim 2, wherein the at least one image processing technique includes at least one of a feature detection algorithm, an edge detection algorithm, a corner detection algorithm, a depth estimation determination algorithm, a focal length estimation algorithm, a camera intrinsics estimation algorithm, a character recognition algorithm, an energy map algorithm, a self-similarity algorithm, a GIST descriptor algorithm, an RGB color space algorithm, a LAB color space algorithm, an HSV color space algorithm, a gray-scale algorithm, a FAST algorithm, a DAISY algorithm, a FREAK algorithm, a SURF algorithm, a BRISK algorithm, an ASR algorithm, an OCR algorithm, and a Gaussian pyramid algorithm.

4. The system of claim 2, wherein the at least one matching method includes at least one of a correlation algorithm, a normalized cross-correlation algorithm, a mutual information algorithm, an FFT algorithm, a histogram matching algorithm, and a Hausdorff distance algorithm.

5. The system of claim 2, wherein the verification engine configured to select a verification technique comprises the verification engine configured to:

select the at least one image processing technique based on one or more characteristics of the at least one candidate result; and

select the at least one matching method based on one or more characteristics of the at least one candidate result.

6. The system of claim 2, wherein each of the at least one image processing technique is associated with a matching method and wherein the verification engine is configured to select a verification technique by selecting at least one image processing technique based on one or more characteristics of the at least one candidate result.

7. The system of claim 2, wherein each of the at least one matching method is associated with an image processing technique and wherein the verification engine is configured to select a verification technique by selecting at least one matching method based on one or more characteristics of the at least one candidate result.

8. The system of claim 1, wherein the plurality of candidate results correspond to candidate images identified for the captured image via an image recognition technique.

9. The system of claim 8, wherein the verification engine is configured to select a verification technique based on one or more characteristics of the at least one candidate result from the plurality of candidate results and the image recognition technique.

10. The system of claim 9, wherein the image recognition technique comprises a SIFT image processing algorithm and a matching algorithm.

11. The system of claim 1, wherein each candidate image from the plurality of candidate images includes at least one of the following information:

an object class, a product class, a product family, a cluster of products, color information, shape information, text information, a logo, a product history, a time, a location, an orientation, a context, a position, a user, a license agreement, an image attribute, a frame rate, a hierarchy, a priority, an ontology, and metadata.

12. The system of claim 1, wherein the verification technique is further selected based on at least one of:

an object class, a product family, a cluster of products, color information, shape information, text information, logos, metadata, a frame rate, a hierarchy, a priority, an ontology, an image attribute, image quality of the captured image, a location of the captured image, time information of the captured image, capture device orientation, environmental conditions, capture device motion, capture device sensor data, a hierarchy, a priority, a user context, a capture device context, and an ontology.

13. The system of claim 1, wherein the verification engine is configured to select a verification technique based on one or more characteristics of the at least one candidate result from the plurality of candidate results and the captured image.

14. The system of claim 1, wherein the verification engine is further configured to:

determine, prior to selecting a verification technique, at least one region of interest for the captured image and each of the plurality of candidate results;

select a verification technique based on the at least one region of interest of at least one candidate result from the plurality of candidate results; and

generate a match score for the at least one candidate result as a function of the at least one region of interest of the captured image, the at least one region of interest of the at least one candidate result, and the selected verification technique.

15. The system of claim 1, wherein the verification engine configured to select a verification technique comprises the verification engine configured to:

calculate a similarity between the plurality of candidate results; and

select the verification technique based on the calculated similarity between the plurality of candidate results.

16. The system of claim 1, wherein the verification engine is further configured to:

apply a rectification algorithm to the captured image, wherein the rectification algorithm is applied prior to the selection of a verification technique.

17. The system of claim 1, wherein the verification engine is further configured to register the plurality of candidate images.

18. The system of claim 1, wherein each of the plurality of candidate images includes a classification attribute identifying a corresponding classification for the candidate image among a plurality of classifications, the system further comprising:

a verification technique database storing a plurality of verification techniques, wherein the verification technique is selected from the plurality of verification techniques;

the verification engine configured to;

process every candidate image in a particular class with each of the plurality of verification techniques;

identify at least one verification technique based on the results of the processing; and

assign a class score to each of the at least one verification technique corresponding to a correlation between the identified at least one verification technique and the particular class.

19. The system of claim 18, wherein the verification engine is further configured to canonize the plurality of candidate images according to the results of the processing.

20. The system of claim 1, wherein:

each of plurality of candidate results includes a confidence score associated with a prior matching of each of the candidate results and the captured image; and

the verification engine is configured to select the verification technique based on the confidence score of the at least one candidate result from the plurality of results.

21. The system of claim 20, wherein the verification engine is further configured to select the verification technique as a function of the confidence score of each of the plurality of candidate results.

22. The system of claim 1, wherein the verification engine is further configured to:

compare the at least one generated match score for the at least one candidate result to a match threshold score;

determine that the at least one generated match score is below the threshold score; and

classify a candidate result corresponding to each of the at least one generated match score as a false positive candidate based on the determination.

23. The system of claim 22, wherein the match threshold score is based on at least one of:

a query image classification, a classification of the at least one candidate result, and the selected verification technique.