Augmented reality in-store product detection system

• US 10,275,825 B2
• Filed: 12/04/2012
• Issued: 04/30/2019
• Est. Priority Date: 12/04/2012
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Inventors: Zises, Matthew Scott
• Primary Examiner(s): Seibert, Christopher B
• Application Number:
  US13/693,126
• Publication Number:
  US 20140156459A1
• Time in Patent Office: 2,338 Days
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers

Abstract

A method and system create, on a portable electronic device, a real-time graphical view of an in-store product display that pinpoints the location of one or more target products on the product display. The graphical view is generated by capturing merchandise display data, such as a video feed of the in-store product display, and processing the data to detect the one or more target products based on predefined product recognition criteria uniquely associated with the respective target products.

Claims

1. A method comprising:

receiving target definition data indicating a shopping list comprising a plurality of target products;

receiving an in-store route that passes at least one of the plurality of target products;

receiving merchandise display data captured by a portable electronic device at a store based on the portable electronic device traversing the in-store route, the merchandise display data indicating physical presence of multiple products on an in-store product display at the store, the merchandise display data including video content of the in-store product display at the store and wireless identifier signals from the multiple products, the wireless identifier signals including location data for at least one of the multiple products;

in an automated operation that is based at least in part on the merchandise display data and that is performed using one or more processor devices configured to perform the automated operation, detecting among the multiple products on the in-store product display a target product forming part of the plurality of target products, the target product detected based on a particular wireless identifier signal of the wireless identifier signals being a unique identifier signal of the target product;

displaying, substantially in real time based on a graphical view of at least a part of the in-store product display being captured by the portable electronic device, an augmented reality video image based on the graphical view, the graphical view including a depiction of a store area where the target product is located;

determining an on-screen position, among the multiple products, of the target product within the graphical view based on particular location data of the particular wireless identifier signal without receiving user input identifying the on-screen position of the target product and while continuing to receive the merchandise display data; and

providing as a part of the graphical view a location indicator that indicates the determined on-screen position, among the multiple products, of the automatically detected target product substantially in real time and in the augmented reality video image.

2. The method of claim 1, wherein the detecting of the target product comprises processing the merchandise display data to identify one or more product recognition criteria for the target product in the merchandise display data.

3. The method of claim 2, further comprising capturing image information by use of an onboard camera of the portable electronic device to provide the merchandise display data, the image information representing an image of at least part of the in-store product display and being included in the video content.

4. The method of claim 3, wherein the detecting of the target product comprises processing the image information to recognize a visual indicator that is unique to the target product.

5. The method of claim 4, wherein the visual indicator comprises one or more trade dress elements of a packaging of the target product.

6. The method of claim 4, wherein the visual indicator comprises a visual product code carried on the target product.

7. The method of claim 3, wherein the location indicator is displayed continuously while the automatically detected target product is visible in the video image.

8. The method of claim 3, further comprising receiving at the portable electronic device the one or more product recognition criteria that define one or more unique product attributes of the target product.

9. The method of claim 8, further comprising transmitting to a store computer system a search query for the target product, the product recognition criteria being received by the portable electronic device from the store computer system responsive to the search query.

10. The method of claim 9, wherein the search query comprises a search image, the method further comprising capturing the search image by use of the onboard camera of the portable electronic device.

11. The method of claim 10, wherein the search query comprises a visual product code unique to the target product, the method further comprising scanning an instance of the visual product code by use of the onboard camera of the portable electronic device.

12. The method of claim 2, wherein the processing of the merchandise display data is to identify any of the plurality of target products in the shopping list, and wherein the method further comprises providing as part of the graphical view a respective location indicator for each of the plurality of target products that are identified.

13. The method of claim 12, further comprising receiving the one or more product recognition criteria for the shopping list from a home pantry system of a user of the portable electronic device.

14. The method of claim 13, further comprising:

monitoring stock levels of products in the home pantry system;

determining that a stock level for a particular product in the home pantry system is below a threshold level;

responsive to the determination, automatically adding the particular product to the shopping list; and

communicating the shopping list to the portable electronic device from the home pantry system by transmitting the one or more product recognition criteria for the respective target products forming part of the shopping list.

15. A system comprising:

a receiver module to receive;

target definition data indicating a shopping list comprising a plurality of target products;

an in-store route that passes at least one of the target products; and

merchandise display data captured by a portable electronic device at a store along the in-store route, the merchandise display data indicating physical presence of multiple products on an in-store product display at the store, the merchandise display data including video content of the in-store product display at the store and wireless identifier signals from the multiple products, the wireless identifier signals including location data for at least one of the multiple products;

a product locator module comprising one or more computer devices configured to automatically detect, based at least in part on the merchandise display data, among the multiple products on the in-store product display a target product forming part of the plurality of target products, the target product detected based on a particular wireless identifier signal of the wireless identifier signals being a unique identifier signal of the target product;

a display module to display, substantially in real time based on a graphical view of at least a part of the in-store product display being captured by the portable electronic device, an augmented reality video image based on the graphical view, the graphical view including a depiction of a store area where the target product is located; and

an indicator module comprising at least one computer processor device configured to;

determine an on-screen position, among the multiple products, of the target product within the graphical view based on particular location data of the particular wireless identifier signal without the system having received any user input that identifies the on-screen position of the target product and while the receiver module continues to receive the merchandise display data;

provide in the graphical view a location indicator that indicates the determined on-screen position, among the multiple products, of the automatically detected target product substantially in real time and in the augmented reality video image.

16. The system of claim 15, wherein the product locator module is configured to process the merchandise display data to identify one or more product recognition criteria for the target product in the merchandise display data.

17. The system of claim 16, wherein the product locator module is configured to process image information captured with an onboard camera of the portable electronic device, the video content including the image information and the image information including one or more images of the in-store product display.

18. The system of claim 17, wherein the product locator module is configured to detect the target product by processing the image information to recognize a visual indicator that is unique to the target product.

19. The system of claim 18, wherein the product locator module is configured to recognize one or more trade dress elements of a packaging of the target product.

20. The system of claim 18, wherein the product locator module is configured to recognize a visual product code carried on the target product.

21. The system of claim 17, wherein the indicator module is configured to dynamically adapt an on-screen position of the location indicator to coincide with the on-screen position of the target product.

22. A non-transitory machine-readable storage medium storing instructions which, in response to being executed by one or more processors of a system, cause the system to perform operations comprising:

receiving target definition data indicating a shopping list comprising a plurality of target products;

receiving an in-store route that passes at least one of the plurality of target products;

receiving merchandise display data captured by a portable electronic device at a store along the in-store route, the merchandise display data indicating physical presence of multiple products on an in-store product display at the store, the merchandise display data including video content of the in-store product display at the store and wireless identifier signals from the multiple products, the wireless identifier signals including location data for at least one of the multiple products;

based at least in part on the merchandise display data, detecting among the multiple products on the in-store product display a target product forming part of the plurality of target products while the portable electronic device is moved along the in-store route, the target product detected based on a particular wireless identifier signal of the wireless identifier signals being a unique identifier signal of the target product;

displaying, substantially in real time based on a graphical view of at least a part of the in-store product display being captured by the portable electronic device, an augmented reality video image based on the graphical view, the graphical view including a depiction of a store area where the target product is located;

determining an on-screen position, among the multiple products, of the target product within the graphical view based on particular location data of the particular wireless identifier signal without receiving user input identifying the on-screen position of the target product and while continuing to receive the merchandise display; and

providing as a part of the graphical view a location indicator that indicates the determined on-screen position, among the multiple products, of the automatically detected target product substantially in real time and in the augmented reality video image.