Image integration search based on human visual pathway model

• US 10,289,927 B2
• Filed: 12/15/2017
• Issued: 05/14/2019
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Adaptive Computation LLC
• Original Assignee: Adaptive Computation LLC
• Inventors: Duong, Tuan A.
• Primary Examiner(s): Woldemariam, Aklilu K
• Application Number:
  US15/843,273
• Publication Number:
  US 20180107873A1
• Time in Patent Office: 515 Days
• Field of Search: 382100, 382118, 382156, 382220, 382294
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Methods, systems, and storage media are disclosed herein for determining objects in image data that match a plurality of captured images, in which one or more of the captured images from among the plurality of captured images may depict only a portion of an object. A computing device may extract a first feature from at least one block of the plurality of blocks from a captured image; determine a second feature; and determine a third feature. The computing device may determine, as a matching image, a stored image from among a plurality of stored images including an object that has greatest maximum correlation with the third feature. The computing device may similarly determine a matching image for each of the captured images of the plurality of images; and determine a final matching image from among the candidate matching images that best matches the combination of the plurality of captured images.

Claims

1. One or more non-transitory computer-readable media comprising instructions, wherein execution of the instructions by a computing device is to cause the computing device to:

receive a search request comprising a first image and a second image;

determine a correlation between at least a first region of interest (ROI) of a first potential matching image from among a plurality of stored images and the first image based on a second low quality feature associated with the first potential matching image and the first image,

wherein the first potential matching image is associated with a first low quality feature determined from the first potential matching image, a first high quality feature determined based on the first low quality feature, and the second low quality feature associated with the first potential matching image determined based on the second high quality feature, and wherein the first high quality feature is larger in size than the first low quality feature or the first high quality feature has a same size as the first low quality feature;

determine another correlation between at least a second ROI of a second potential matching image from among the plurality of stored images and the second image based on a another second low quality feature associated with the second potential matching image and the second image,

wherein the second potential matching image is associated with another first low quality feature determined from the second potential matching image, another first high quality feature determined based on the other first low quality feature of the second potential matching image, and the other second low quality feature associated with the second potential matching image determined based on the other first high quality feature of the second potential matching image, and wherein the other first high quality feature is larger in size than the other first low quality feature or the other first high quality feature has a same size as the other first low quality feature; and

determine, as query results, a result image from among the first potential matching image and the second potential matching image that has maximum correlation to both the first image and the second image.

2. The one or more non-transitory computer-readable media of claim 1, wherein the first image or the second image depicts only a portion of an entire object and the result image depicts the entire object.

3. The one or more non-transitory computer-readable media of claim 2, wherein the entire object comprises a product or a human face.

4. The one or more non-transitory computer-readable media of claim 1, wherein the first low quality feature determined from the first potential matching image is a vertebrate feature, the first high quality feature is a fovea feature, and the second low quality feature associated with the first potential matching image is a lateral geniculate nucleus (LGN) feature, and wherein the other first low quality feature determined from the second potential matching image is another vertebrate feature, the other first high quality feature is another fovea feature, and the other second low quality feature is another LGN feature.

5. The one or more non-transitory computer-readable media of claim 1, wherein:

the first low quality feature is a more blurry feature than the first high quality feature or pixels in the first low quality feature have a larger size than pixels in the first high quality feature

the second low quality feature associated with the first potential matching image is a more blurry version of the first high quality feature or pixels in the second low quality feature have a larger size than pixels in the first high quality feature,

the other first low quality feature is a more blurry feature than the other first high quality feature or pixels in the other first low quality feature have a larger size than pixels in the other first high quality feature, and

the other second low quality feature is a more blurry version of the other first high quality feature or pixels in the other second low quality feature have a larger size than pixels in the other first high quality feature.

6. The one or more non-transitory computer-readable media of claim 1, wherein execution of the instructions is to cause the computing device to:

generate a plurality of blocks from the first image or the second image, wherein each block of the plurality of blocks is representative of a corresponding region of the first image or the second image;

extract a first low quality feature of the first image or the second image from at least one block of the plurality of blocks;

determine a first high quality feature of the image based at least on the first low quality feature of the first image or the second image; and

determine a second low quality feature of the first image or the second image based on the first high quality feature of the first image or the second image;

control storage of the first image or the second image in association with the first low quality feature, the first high quality feature, and the second low quality feature, wherein the plurality of stored images is to include the first image or the second image upon storage of the first image or the second image.

7. The one or more non-transitory computer-readable media of claim 1, wherein execution of the instructions is to cause the computing device to:

for each of a plurality of different scaled versions of the plurality of stored images,

generate a plurality of blocks from a scaled image of the plurality of different scaled versions of the plurality of stored images, wherein each block of the plurality of blocks is representative of a corresponding region of the scaled image;

extract a first low quality feature of the scaled image from at least one block of the plurality of blocks;

determine a first high quality feature of the scaled image based at least on the first low quality feature of the scaled image; and

determine a second low quality feature of the scaled image based on the first high quality feature of the scaled image.

8. The one or more non-transitory computer-readable media of claim 1, wherein execution of the instructions is to cause the computing device to:

generate a plurality of scaled first images based on the first object, each scaled first image of the plurality of scaled first images associated with a particular scale factor;

generate a plurality of scaled second images based on the second object, each scaled second image of the plurality of scaled second images associated with the particular scale factor;

wherein, to determine the correlation between at least the first ROI of the first potential matching image and the first object, execution of the instructions is to cause the computing device to;

determine the correlation based on the second low quality feature associated with the first potential matching image and the scaled first image from among the plurality of scaled first images; and

wherein, to determine the other correlation between at least the second ROI of the second potential matching image and the second object, execution of the instructions is to cause the computing device to;

determine other the correlation based on the second low quality feature associated with the second potential matching image and the scaled second image from among the plurality of scaled second images.

9. The one or more non-transitory computer-readable media of claim 1, wherein the query results are determined in accordance with a neural network approximation function.

10. The one or more non-transitory computer-readable media of claim 1, wherein execution of the instructions is to cause the computing device to:

determine, for the first image, a first set of features comprising a low resolution feature, a high resolution feature, and a low resolution version of the high resolution feature; and

Wherein, to determine the correlation between at least the first ROI of the first potential matching image and the first image, execution of the instructions is to cause the computing device to;

determine the correlation based on the first set of features associated with the first image and the second low quality feature associated with the first potential matching image.

11. A computing system comprising:

communication circuitry arranged to communicate with a remote storage system and one or more client devices, wherein the storage system is arranged to store a plurality of stored images, and the communication circuitry is arranged to receive a query from a client device of the one or more client devices, wherein the query comprises an image including a first object and a second object, and send query results to the client device in response to the query request; and

processor circuitry communicatively coupled with the communication circuitry, wherein the processor circuitry is arranged to;

determine a correlation between at least a first region of interest (ROI) of a first potential matching image from among the plurality of stored images and the first object based on a second low quality feature associated with the first potential matching image and the first object,

wherein the first potential matching image is associated with a first low quality feature determined from the first potential matching image, a first high quality feature determined based on the first low quality feature, and the second low quality feature associated with the first potential matching image determined based on the first high quality feature, and wherein the first high quality feature of the first potential matching is larger in size than the first low quality feature determined from the first potential matching image or the first high quality feature of the first potential matching has a same size as the first low quality feature from the first potential matching image,

determine another correlation between at least a second region of interest (ROI) of a second potential matching image from among the plurality of stored images and the second object based on a second low quality feature associated with the second potential matching image and the second object,

wherein the second potential matching image is associated with a first low quality feature determined from the second potential matching image, a first high quality feature determined based on the first low quality feature of the second potential matching image, and the second low quality feature associated with the second potential matching image determined based on the first high quality feature of the second potential matching image, and wherein the first high quality feature of the second potential matching is larger in size than the first low quality feature determined from the second potential matching image or the first high quality feature of the second potential matching has a same size as the first low quality feature from the second potential matching image,

determine, as the query results, a result image from among the first potential matching image and the second potential matching image that has maximum correlation to both the first object and the second object, wherein the maximum correlation is a greater one among the correlation and the other correlation, and

provide the query results to the communication circuitry to be sent to the client device in response to the query request.

12. The computing system of claim 11, wherein the first object depicts only a portion of an entire object and the image determined to be the query results depicts the entire object.

13. The computing system of claim 12, wherein the object comprises a product or a face.

14. The computing system of claim 11, wherein the first low quality feature determined from the first potential matching image or the second potential matching image is a vertebrate feature, the first high quality feature of the first potential matching image or the second potential matching image is a fovea feature, and the second low quality feature associated with the first potential matching image or the second potential matching image is a lateral geniculate nucleus (LGN) feature.

15. The computing system of claim 11, wherein:

the first low quality feature determined from the first potential matching image is a more blurry feature than the first high quality feature of the first potential matching image or pixels in the first low quality feature have a larger size than pixels in the first high quality feature of the first potential matching image,

the first low quality feature determined from the second potential matching image is a more blurry feature than the first high quality feature of the second potential matching image or pixels in the first low quality feature of the second potential matching image have a larger size than pixels in the first high quality feature of the second potential matching image,

the second low quality feature associated with the first potential matching image is a more blurry version of the first high quality feature of the first potential matching image or pixels in the second low quality feature have a larger size than pixels in the first high quality feature of the first potential matching image, and

the second low quality feature associated with the second potential matching image is a more blurry version of the first high quality feature of the second potential matching image or pixels in the second low quality feature have a larger size than pixels in the first high quality feature of the second potential matching image.

16. The computing system of claim 11, wherein the image is a captured image, and the processor circuitry is to:

generate a plurality of blocks from the captured image, wherein each block of the plurality of blocks is representative of a corresponding region of the captured image,

extract a first low quality feature of the captured image from at least one block of the plurality of blocks,

determine a first high quality feature of the captured image based at least on the first low quality feature of the captured image, and

determine a second low quality feature of the captured image based on the first high quality feature of the captured image, and

Wherein the communication circuitry is to send the first low quality feature, the first high quality feature, and the second low quality feature to the remote storage system for storage in association with the captured image, wherein the plurality of stored images includes the captured image upon storage of the captured image.

17. The computing system of claim 11, wherein, for each of a plurality of different scaled versions of the plurality of stored images, the processor circuitry is to:

generate a plurality of blocks from a scaled image from the plurality of different scaled versions of the plurality of stored images, wherein each block of the plurality of blocks is representative of a corresponding region of the scaled image,

extract a first low quality feature of the scaled image from at least one block of the plurality of blocks,

determine a first high quality feature of the scaled image based at least on the first low quality feature of the scaled image, and

determine a second low quality feature of the scaled image based on the first high quality feature of the scaled image.

18. The computing system of claim 11, wherein the processor circuitry is to:

generate a plurality of scaled first images based on the first object, each scaled first image of the plurality of scaled first images associated with a particular scale factor,

generate a plurality of scaled second images based on the second object, each scaled second image of the plurality of scaled second images associated with the particular scale factor,

wherein of the processor circuitry is to determine the correlation between at least the first ROI of the first potential matching image and the first object based on the second low quality feature associated with the first potential matching image and the scaled first image from among the plurality of scaled first images, and

wherein of the processor circuitry is to determine the other correlation between at least the second ROI of the second potential matching image and the second object based on the second low quality feature associated with the second potential matching image and the scaled second image from among the plurality of scaled second images.

19. The computing system of claim 11, wherein the processor circuitry is to determine the query results in accordance with a neural network approximation function.

20. The computing system of claim 11, wherein the processor circuitry is to:

determine, for the first object, a first set of features comprising a low resolution feature, a high resolution feature, and a low resolution version of the high resolution feature, and

wherein of the processor circuitry is to determine the correlation between at least the first ROI of the first potential matching image and the first object based on the first set of features associated with the first object and the second low quality feature associated with the first potential matching image.