Visual search in a controlled shopping environment

• US 9,424,598 B1
• Filed: 12/02/2013
• Issued: 08/23/2016
• Est. Priority Date: 12/02/2013
• Status: Expired due to Fees

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Kraft, Adam Wiggen
• Primary Examiner(s): Dunham, Jason
• Assistant Examiner(s): O'SULLIVAN, KATHERINE L
• Application Number:
  US14/094,655
• Time in Patent Office: 995 Days
• Field of Search: 705 261- 272
• US Class Current: 1/1
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0627 using item specifications
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/025 using location based information parameters

Abstract

Various embodiments enable a customer to quickly search additional information (e.g., product variations, sizes, price and availability) related to a specific product. For example, the customer can request additional information of a specific product by submitting an image of the specific product from a computing device. In one embodiment, location of the customer can be determined based on the image submitted by the customer. Product features can be extracted from the image according to various imaging processing and text recognition algorithms and then used to match products that are within view of the customer. A search results with additional information of the specific product can be provided to the computing device for presentation to the customer.

Claims

1. A system for providing product information for a specified product to a customer in a store, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

receive, from a computing device associated with the customer, an image of the specified product;

track a location of the customer in the store by comparing a background light pattern of the image to ambient light patterns detected through sensors placed throughout the store, the ambient light patterns corresponding to a section of the store, the section of the store including a type or brand of product;

locate, in the image, one or more features of the specified product;

match the one or more features of the specified product against one or more features of each of a plurality of products in the store, wherein the plurality of products is selected based at least in part on proximity of the products to the location of the customer; and

transmit the product information for at least one matched product to the computing device to be presented to the customer, wherein the product information includes at least price, size and availability of the at least one matched product in the store.

2. The system of claim 1, wherein the instructions when executed further cause the system to:

extract features from the background of the image; and

pinpoint the location of the customer by matching the extracted features against in-store visual information for the store.

3. The system of claim 1, wherein the instructions when executed further cause the system to:

update the location of the specified product by at east taking into account the determined location of the customer when the specified product is matched.

4. The system of claim 1, wherein a management server of the store is configured to dynamically update the product information on a needed basis by taking into account products that have been checked out by other customers in the store and put into shopping carts by other customers in the store.

19. The system of claim 1, wherein the detected light patterns are pre-learnt.

20. The system of claim 1, wherein the detected light patterns are dynamically learned.

5. A computer-implemented method, comprising:

receiving an image of a specified product from a computing device of a customer in a store;

determining a location of the customer in the store by comparing a background light pattern of the image to ambient light patterns detected through sensors placed throughout the store, the ambient light patterns corresponding to a section of the store, the section of the store including a type or brand of product;

matching one or more characteristics of one or more objects in the image against features of one or more products offered via the store, wherein the one or more products are selected based at least in part on proximity of the products to the location of the customer; and

transmitting product information for at least one matched product to the computing device to be presented to the customer.

6. The computer-implemented method of claim 5, further comprising:

analyzing sensor data from the computing device to extract geolocation data; and

determining the location of the customer by matching the geolocation data against GPS coordinates of the store.

7. The computer-implemented method of claim 5, further comprising:

calibrating GPS coordinates of at least three Wi-Fi and/or Bluetooth transmitters placed throughout the store;

receiving, by the at least three Wi-Fi and/or Bluetooth transmitters, signals from the computing device;

analyzing signal strength received by the at least three Wi-Fi and/or Bluetooth transmitters; and

determining the location of the customer by using one or more triangulation algorithms to calculate distances from the computing device to the at least three Wi-Fi and/or Bluetooth transmitters.

8. The computer-implemented method of claim 5, further comprising:

analyzing sensor data from the computing device to extract sound waves at the location of the customer; and

determining the location of the customer by matching the sound waves against pre-learnt sound patterns at specific locations in the stores.

9. The computer-implemented method of claim 5, further comprising:

receiving, from the computing device, one or more voice commands of the customer;

analyzing the one or more voice commands to extract information corresponding to the location of the customer; and

combining the information extracted from the one or more voice commands in the determining process of the location of the customer.

10. The computer-implemented method of claim 5, further comprising:

selecting one or more algorithms to be applied in the matching process at least based on one or more product categories in the proximity of the customer'"'"'s location.

11. The computer-implemented method of claim 10, further comprising:

pre-loading the product information of the one or more products and the one or more algorithms onto the computing device to maximize product searching experience for the customer;

matching, by the computing device, the one or more characteristics of the one or more objects in the image against the features of the one or more products; and

presenting, by the computing device, the product information of the at least one matched product to the customer.

12. The computer-implemented method of claim 5, further comprising:

extracting one or more features from the background of the image; and

determining the location of the customer by matching extracted background features against in-store visual information for the store.

13. The computer-implemented method of claim 12, further comprising:

determining a specific 3D area that the customer is viewing when taking the image by analyzing the extracted background features, or sensor data from the computing device when the customer was taking the image; and

limiting the one or more products in the matching process to products in the specific 3D area.

14. The computer-implemented method of claim 5, further comprising:

enabling the customer to select a refinement to the product information presented, the refinement corresponding to a selected product attribute, wherein the product information includes at least price, size and availability of the at least one matched product in the store.

15. The computer-implemented method of claim 5, further comprising:

updating product information in the store by at least taking into account inventory information of the one or more products in the proximity of the customer'"'"'s location, the inventory information of the one or more products including at least one product matched or found missing in the matching process.

16. The computer-implemented method of claim 15, wherein the image of the specified product is saved for future matching against images taken by other customers in the store.

21. The computer-implemented method of claim 5, wherein the detected light patterns are pre-learnt.

22. The computer-implemented method of claim 5, wherein the detected light patterns are dynamically learned.

17. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to:

receive an image of a specified product from a computing device of a customer;

match one or more features of the specified product against features of one or more products currently offered via a store;

transmit product information for at least one matched product to the computing device to be presented to the customer; and

determine a location of the customer in the store by comparing a background light pattern of the image to ambient light patterns detected through sensors placed throughout the store, the ambient light patterns corresponding to a section of the store, the section of the store including a type or brand of product.

18. The non-transitory computer-readable storage medium of claim 17, wherein the product information includes at least price, size and availability of the at least one matched product in the store, and wherein the specified product has no bar code.

23. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the computing system to:

determine a product category located within a proximity of a customer'"'"'s location, the product category being associated with a matching algorithm; and

select the matching algorithm to be applied in the matching process based at least in part on the product category.

24. The non-transitory computer-readable storage medium of claim 23, wherein the instructions when executed further cause the computing system to:

pre-load product information of the one or more products and the matching algorithm onto the computing device to maximize product searching experience for the customer, wherein the one or more products are selected based at least in part on proximity of the products to the location of the customer.