Hybrid detection recognition system

• US 10,242,036 B2
• Filed: 06/30/2016
• Issued: 03/26/2019
• Est. Priority Date: 08/14/2013
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Kwon, Junghyun, Narasimha, Ramya, Schwartz, Edward L., McFarland, Max, Savarese, Silvio, Berkner, Kathrin
• Primary Examiner(s): Wilder, A. Hunter
• Application Number:
  US15/199,553
• Publication Number:
  US 20160342863A1
• Time in Patent Office: 999 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data

Abstract

A system and method for determining an object or product represented in an image is disclosed. The system receives a first image, determines a region of interest in the first image, determines a classification score for the region of interest using a convolutional neural network that assigns the region of interest the classification score corresponding to a class, and identifies a first product in the first image based on the classification score.

Claims

1. A method comprising:

receiving, by one or more processors, a first image;

determining, by the one or more processors, a plurality of regions of interest that share a similar spatial location in the first image;

ranking, by the one or more processors, the plurality of regions of interest;

selecting, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;

determining, by the one or more processors, classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;

matching, by the one or more processors, the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;

adjusting, by the one or more processors, the classification scores based on the matching score; and

identifying, by the one or more processors, a first product in the first image based on the adjusted classification scores.

2. The method of claim 1, wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

3. The method of claim 1, further comprising:

determining a first classification score for a first region of interest in the first image;

determining a second classification score for a second region of interest in the first image;

determining whether the first region of interest and the second region of interest share the similar spatial location; and

responsive to the first region of interest and the second region of interest sharing the similar spatial location, combining the first classification score and the second classification score to determine a result class for the similar spatial location.

4. The method of claim 1, wherein matching the predetermined number of regions of interest in the first image to the indexed images using the model-based features comprises:

extracting a first feature of the first image;

determining whether the first feature of the first image matches a second feature of the second image;

determining whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and

responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identifying the shape formed by the first feature of the first image as one of the predetermined number of regions of interest.

5. A system comprising:

one or more processors; and

a memory, the memory storing instructions, which when executed cause the one or more processors to;

receive a first image;

determine a plurality of regions of interest that share a similar spatial location in the first image;

rank the plurality of regions of interest;

select, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;

determine classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;

match the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;

adjust the classification scores based on the matching score; and

identify a first product in the first image based on the adjusted classification scores.

6. The system of claim 5, wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

7. The system of claim 5, wherein the instructions further cause the one or more processors to:

determine a first classification score for a first region of interest in the first image;

determine a second classification score for a second region of interest in the first image;

determine whether the first region of interest and the second region of interest share the similar spatial location; and

responsive to the first region of interest and the second region of interest sharing the similar spatial location, combine the first classification score and the second classification score to determine a result class for the similar spatial location.

8. The system of claim 5, wherein to match the predetermined number of regions of interest in the first image to the indexed images using the model-based features, the instructions further cause the one or more processors to:

extract a first feature of the first image;

determine whether the first feature of the first image matches a second feature of the second image;

determine whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and

responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identify the shape formed by the first feature of the first image as one of the predetermined number of regions of interest.

9. A computer program product comprising a non-transitory computer readable medium storing a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a first image;

determine a plurality of regions of interest that share a similar spatial location in the first image;

rank the plurality of regions of interest;

select, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;

determine classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;

match the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;

adjust the classification scores based on the matching score; and

identify a first product in the first image based on the adjusted classification scores.

10. The computer program product of claim 9, wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

11. The computer program product of claim 9, wherein the computer readable program when executed on the computer further causes the computer to:

determine a first classification score for a first region of interest in the first image;

determine a second classification score for a second region of interest in the first image;

determine whether the first region of interest and the second region of interest share the similar spatial location; and

responsive to the first region of interest and the second region of interest sharing the similar spatial location, combine the first classification score and the second classification score to determine a result class for the similar spatial location.

12. The computer program product of claim 9, wherein to match the predetermined number of regions of interest in the first image to the indexed images using the model-based features, the computer readable program when executed on the computer further causes the computer to:

extract a first feature of the first image;

determine whether the first feature of the first image matches a second feature of the second image;

determine whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and

responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identify the shape formed by the first feature of the first image as one of the predetermined number of regions of interest.