IMAGE INTEGRATION SEARCH BASED ON HUMAN VISUAL PATHWAY MODEL

• US 20180107873A1
• Filed: 12/15/2017
• Published: 04/19/2018
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Adaptive Computation LLC
• Original Assignee: Adaptive Computation LLC
• Inventors: Duong, Tuan A.
• Granted Patent: US 10,289,927 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Methods, systems, and storage media are disclosed herein for determining objects in image data that match a plurality of captured images, in which one or more of the captured images from among the plurality of captured images may depict only a portion of an object. A computing device may extract a first feature from at least one block of the plurality of blocks from a captured image; determine a second feature; and determine a third feature. The computing device may determine, as a matching image, a stored image from among a plurality of stored images including an object that has greatest maximum correlation with the third feature. The computing device may similarly determine a matching image for each of the captured images of the plurality of images; and determine a final matching image from among the candidate matching images that best matches the combination of the plurality of captured images.

Claims

1. One or more non-transitory computer-readable media having instructions that, when executed, cause a computing device to:

receive a query request comprising a first image and a second image;

determine a correlation between at least a first region of interest (ROI) of a first potential matching image, from among a plurality of stored images, and the first image based on a third feature associated with the first potential matching image and the first image,

wherein the first potential matching image is associated with a first feature determined from the first potential matching image, a second feature determined based on the first feature, and the third feature associated with the first potential matching image determined based on the second feature;

determine a correlation between at least a second region of interest (ROI) of a second potential matching image, from among the plurality of stored images, and the second image based on a third feature associated with the second potential matching image and the second image,

wherein the second potential matching image is associated with a first feature determined from the second potential matching image, a second feature determined based on the first feature of the second potential matching image, and the third feature associated with the second potential matching image determined based on the second feature of the second potential matching image; and

determine, as query results, an image from among the first potential matching image and the second potential matching image that has maximum correlation to both the first image and the second image.

2. The one or more non-transitory computer-readable media of claim 1, wherein the first image depicts only a portion of an object and the image determined to be the query results depicts the object.

3. The one or more non-transitory computer-readable media of claim 2, wherein the object comprises a product or a face.

4. The one or more non-transitory computer-readable media of claim 1, wherein the first feature determined from the first potential matching image comprises a vertebrate feature, the second feature comprises a fovea feature, and the third feature associated with the first potential matching image comprises a lateral geniculate nucleus (LGN) feature.

5. The one or more non-transitory computer-readable media of claim 1, wherein the first feature determined from the first potential matching image comprises a low resolution feature, the second feature comprises a high resolution feature, and the third feature associated with the first potential matching image comprises a low resolution version of the second feature.

6. The one or more non-transitory computer-readable media of claim 1, wherein the instructions further cause the computing device, in response to execution of the instructions by the computing device, to:

generate a plurality of blocks from a captured image, wherein each block of the plurality of blocks is representative of a corresponding region of the captured image;

extract a first feature of the captured image from at least one block of the plurality of blocks;

determine a second feature of the captured image based at least on the first feature of the captured image; and

determine a third feature of the captured image based on the second feature of the captured image;

wherein the captured image comprises each stored image of the plurality of stored images.

7. The one or more non-transitory computer-readable media of claim 6, wherein the instructions further cause the computing device, in response to execution of the instructions by the computing device, to:

perform the generate the plurality of blocks, extract the first feature, determine the second feature, and determine the third feature for each of a plurality of different scaled versions of the plurality of stored images.

8. The one or more non-transitory computer-readable media of claim 1, wherein the instructions further cause the computing device, in response to execution of the instructions by the computing device, to:

generate a plurality of scaled first images based on the first image, each scaled first image of the plurality of scaled first images associated with a particular scale factor;

generate a plurality of scaled second images based on the second image, each scaled second image of the plurality of scaled second images associated with the particular scale factor;

wherein determine the correlation between at least the first ROI of the first potential matching image and the first image comprises determining the correlation based on the third feature associated with the first potential matching image and the scaled first image from among the plurality of scaled first images; and

wherein determine the correlation between at least the second ROI of the second potential matching image and the second image comprises determining the correlation based on the third feature associated with the second potential matching image and the scaled second image from among the plurality of scaled second images.

9. The one or more non-transitory computer-readable media of claim 1, wherein the query results is determined in accordance with a neural network approximation function.

10. The one or more non-transitory computer-readable media of claim 1, wherein the instructions further cause the computing device, in response to execution of the instructions by the computing device, to:

determine, for the first image, a first set of features comprising a low resolution feature, a high resolution feature, and a low resolution version of the high resolution feature; and

wherein to determine the correlation between at least the first ROI of the first potential matching image and the first image comprises determining the correlation based on the first set of features associated with the first image and the third feature associated with the first potential matching image.

11. A computing device comprising:

at least one storage device including a plurality of stored images; and

at least one processor in communication with the at least one storage device, wherein the at least one processor;

receives a query request comprising a first image and a second image,

determines a correlation between at least a first region of interest (ROI) of a first potential matching image, from among the plurality of stored images, and the first image based on a third feature associated with the first potential matching image and the first image,

wherein the first potential matching image is associated with a first feature determined from the first potential matching image, a second feature determined based on the first feature, and the third feature associated with the first potential matching image determined based on the second feature,

determines a correlation between at least a second region of interest (ROI) of a second potential matching image, from among the plurality of stored images, and the second image based on a third feature associated with the second potential matching image and the second image,

wherein the second potential matching image is associated with a first feature determined from the second potential matching image, a second feature determined based on the first feature of the second potential matching image, and the third feature associated with the second potential matching image determined based on the second feature of the second potential matching image,

determines, as query results, an image from among the first potential matching image and the second potential matching image that has maximum correlation to both the first image and the second image, and

provides the query results in response to the query request.

12. The computing device of claim 11, wherein the first image depicts only a portion of an object and the image determined to be the query results depicts the object.

13. The computing device of claim 12, wherein the object comprises a product or a face.

14. The computing device of claim 11, wherein the first feature determined from the first potential matching image comprises a vertebrate feature, the second feature comprises a fovea feature, and the third feature associated with the first potential matching image comprises a lateral geniculate nucleus (LGN) feature.

15. The computing device of claim 11, wherein the first feature determined from the first potential matching image comprises a low resolution feature, the second feature comprises a high resolution feature, and the third feature associated with the first potential matching image comprises a low resolution version of the second feature.

16. The computing device of claim 11, wherein the at least one processor:

generates a plurality of blocks from a captured image, wherein each block of the plurality of blocks is representative of a corresponding region of the captured image,

extracts a first feature of the captured image from at least one block of the plurality of blocks,

determines a second feature of the captured image based at least on the first feature of the captured image,

determines a third feature of the captured image based on the second feature of the captured image, and

wherein the captured image comprises each stored image of the plurality of stored images.

17. The computing device of claim 16, wherein the at least one processor:

performs generation of the plurality of blocks, extraction of the first feature, determination of the second feature, and determination of the third feature for each of a plurality of different scaled versions of the plurality of stored images.

18. The computing device of claim 11, wherein the at least one processor:

generates a plurality of scaled first images based on the first image, each scaled first image of the plurality of scaled first images associated with a particular scale factor,

generates a plurality of scaled second images based on the second image, each scaled second image of the plurality of scaled second images associated with the particular scale factor,

wherein determination of the correlation between at least the first ROI of the first potential matching image and the first image comprises determining the correlation based on the third feature associated with the first potential matching image and the scaled first image from among the plurality of scaled first images, and

wherein determination of the correlation between at least the second ROI of the second potential matching image and the second image comprises determining the correlation based on the third feature associated with the second potential matching image and the scaled second image from among the plurality of scaled second images.

19. The computing device of claim 11, wherein the query results is determined in accordance with a neural network approximation function.

20. The computing device of claim 11, wherein the at least one processor:

determines, for the first image, a first set of features comprising a low resolution feature, a high resolution feature, and a low resolution version of the high resolution feature, and

wherein determination of the correlation between at least the first ROI of the first potential matching image and the first image comprises determining the correlation based on the first set of features associated with the first image and the third feature associated with the first potential matching image.