Computer vision product recognition

• US 10,521,844 B2
• Filed: 11/25/2014
• Issued: 12/31/2019
• Est. Priority Date: 11/25/2014
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: Gupta, Ankit, Shah, Saurav, Sowdaboina, Venkata Pranay Kumar
• Primary Examiner(s): Misiaszek, Michael
• Application Number:
  US14/553,062
• Publication Number:
  US 20160148292A1
• Time in Patent Office: 1,862 Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G06Q 30/0639 Item locations

Abstract

A system for computer-aided visual recognition of products may be used by a customer operating a mobile computing device having a camera. A customer can direct the camera at a targeted product for which the customer desires to purchase or learn information. Image recognition operations can be carried out to compare the captured image against images from a prepopulated product image store of known products. Upon a positive match, information regarding the identified product may be gathered and transmitted to the customer and displayed on the mobile computing device. The customer may be presented with various options regarding the identified product, including adding the product to a shopping list, a to-do list, a wish list, or other types of lists.

Claims

1. A computer-implemented method for identifying a product, comprising:

receiving, at a processing device of a product recognition server, one or more signals from a mobile computing device corresponding to consecutively captured images of a targeted product or a packaging of the targeted product, wherein the targeted product comprises the targeted product offered for sale by a retailer or recently consumed by a user, or an advertisement of the targeted product, wherein the mobile computing device comprises smart glasses;

analyzing, at the processing device, the consecutively captured images of the targeted product by comparing each image of the consecutively captured images to a plurality of images of retail products offered for sale by the retailer;

receiving, with the processing device, a match score between the consecutively captured images of the targeted product and the plurality of images of retail products offered by the retailer;

when the match score is lower than a pre-determined minimum match score threshold, determining, at the processing device, that the consecutively captured images of the targeted product does not match the plurality of images of retail products and that the retailer does not offer the targeted product for sale; and

when the match score is higher than the pre-determined minimum match score threshold;

determining, at the processing device, that the consecutively captured images of the targeted product matches at least one image of the plurality of images of retail products offered for sale by the retailer;

confirming, at the processing device, a product selection of the targeted product after identifying the targeted product within a pre-determined number of the consecutively captured images to minimize false positive identifications of the targeted product;

obtaining, from a product database, information associated with the targeted product;

transmitting, from the processing device, the information to the smart glasses by a graphical user interface within a field of view of the user; and

displaying, on a display of the smart glasses, by the graphical user interface, at least one of a text, graphic, or video including the information.

2. The method of claim 1, wherein the information comprises a retail price of the targeted product.

3. The method claim 1, wherein the information comprises an availability status of the targeted product at a selected retail store.

4. The method of claim 3, wherein the selected retail store is selected based on physical proximity to the mobile computing device.

5. The method of claim 1, wherein analyzing, at the processing device, the consecutively captured images of the targeted product to the plurality of images of retail products further comprises retrieving the plurality of images of retail products from a product image store.

6. The method of claim 1, wherein analyzing, at the processing device, the consecutively captured images of the targeted product to the plurality of images of retail products further comprises applying a feature detection algorithm to the consecutively captured images of the targeted product.

7. The method of claim 6, wherein applying the feature detection algorithm to the consecutively captured images of the targeted product comprises carrying out a Speeded-Up Robust Features (SURF) feature detection algorithm.

8. The method of claim 1, wherein analyzing, at the processing device the consecutively captured images of the targeted product to the plurality of images of retail products further comprises disregarding multiple ones of the consecutively captured images when an undesired product is captured by the consecutively captured images.

9. The method of claim 1, further comprising:

receiving from the mobile computing device, a signal of the one or more signals indicating an input to add the targeted product to a list; and

transmitting an instruction to a customer database to add the targeted product to a list associated with an operator of the mobile computing device.

10. The method of claim 9, wherein the list comprises a shopping list.

11. A method for identifying a product, comprising:

receiving, at a processing device or a product recognition server, one or more signals from a camera of a mobile computing device corresponding to capturing consecutively captured images of a targeted product or a packaging of the targeted product, wherein the targeted product comprises the targeted product offered for sale by a retailer or recently consumed by a user, or an advertisement of the targeted product, wherein the mobile computing device comprises smart glasses;

analyzing, at the processing device, the consecutively captured images of the targeted product by comparing each image of the consecutively captured images to a plurality of images of retail products offered for sale by the retailer;

receiving, with the processing device, a match score between the consecutively captured images of the targeted product and the plurality of images of retail products offered by the retailer;

when the match score is lower than a pre-determined minimum match score threshold, determining, at the processing device, that the consecutively captured images of the targeted product does not match the plurality of images of retail products and that the retailer does not offer the targeted product for sale; and

when the match score is higher than the pre-determined minimum match score threshold;

determining, at the processing device, that the consecutively captured images of the targeted product matches at least one image of the plurality of images of retail products offered for sale by the retailer;

confirming, at the processing device, a product selection of the targeted product after identifying the targeted product within a pre-determined number of the consecutively captured images to minimize false positive identifications of the targeted product;

obtaining, from a product database, information associated with the targeted product;

transmitting, from the processing device, the information to the smart glasses by a graphical user interface within a field of view of the user; and

displaying on a display of the smart glasses, by the graphical user interface, at least one of a text, graphic, or video including the information.

12. A computer-implemented system for identifying a product comprising:

one or more processors;

a product recognition server comprising a memory device and a processing device, the memory device storing computer-readable instructions directing the processing device to;

receive one or more signals from a mobile computing device corresponding to consecutively captured images of a targeted product or a packaging of the targeted product, wherein the targeted product comprises the targeted product offered for sale by a retailer or recently consumed by a user, or an advertisement of the targeted product, wherein the mobile computing device comprises smart glasses;

analyze the consecutively captured images of the targeted product by comparing each image of the consecutively captured images to a plurality of images of retail products offered for sale by the retailer;

receive a match score between the consecutively captured images of the targeted product and the plurality of images of retail products offered by the retailer;

when the match score is lower than a pre-determined minimum match score threshold, determine that the consecutively captured images of the targeted product does not match the plurality of images of retail products and that the retailer does not offer the targeted product for sale; and

when the match score is higher than the pre-determined minimum match score threshold;

determine that the consecutively captured images of the targeted product matches at least one image of the plurality of images of retail products offered by the retailer for sale by the retailer;

confirm a product selection of the targeted product after identifying the targeted product within a pre-determined number of the consecutively captured images to minimize false positive identifications of the targeted product;

obtain, from a product database, information associated with the targeted product;

transmit the information to the smart glasses by a graphical user interface within a field of view of the user; and

display, on a display of the smart glasses, by the graphical user interface, at least one of a text, graphic, or video including the information.

13. The computer-implemented system of claim 12, wherein the information comprises a retail price of the targeted product.

14. The computer-implemented system of claim 12, wherein the information comprises an availability status of the targeted product at a selected retail store.

15. The computer-implemented system of claim 14, wherein the selected retail store is selected based on physical proximity to the mobile computing device.

16. The computer-implemented system of claim 12, further comprising a product image store.

17. The computer-implemented system of claim 16, wherein the computer-readable instructions further direct the processing device to retrieve the plurality of images of retail products from the product image store.

18. The computer-implemented system of claim 12, wherein the computer-readable instructions further direct the processing device to apply a feature detection algorithm to the consecutively captured images of the targeted product.

19. The computer-implemented system of claim 12, wherein the computer-readable instructions further direct the processing device to disregard multiple ones of the consecutively captured images when an undesired product is captured.

20. The computer-implemented system of claim 12, further comprising a customer database, wherein the computer-readable instructions further direct the processing device to:

receive, from the mobile computing device, a signal of the one or more signals indicating an input to add the targeted product to a list; and

transmit an instruction to the customer database to add the targeted product to a list associated with an operator of the mobile computing device.