Machine learning methods and systems for managing retail store processes involving cashier-less transactions

• US 10,510,219 B1
• Filed: 09/20/2018
• Issued: 12/17/2019
• Est. Priority Date: 07/25/2015
• Status: Active Grant

• Current Assignee: Albert S. Penilla, Gary M. Zalewski
• Original Assignee: Albert S. Penilla, Gary M. Zalewski
• Inventors: Zalewski, Gary M., Penilla, Albert S.
• Primary Examiner(s): Chen, Alan
• Application Number:
  US16/137,498
• Time in Patent Office: 453 Days
• Field of Search: None
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/327 Short range or proximity payments by means of M-devices
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G07G 1/0072 with means for detecting the weight of the article of which the code is read, for the verification of the registration weighing combined with recording means, e.g. in weighing scale, G01G19/415

Abstract

Devices, systems, and method are provided for identifying actions in a retail store. One method includes sampling a shopping environment using one or more sensors. The method includes receiving output of the sampling as feature inputs to a machine learning classifier model to derive one or more labels characterizing a state of an item. At least one processing entity associated with the retail store detects the state of the item to change from one as item taken to one as item returned, and sensor data from said one or more sensors characterizing the item as having been returned to a wrong location in the retail store. In further embodiments, items taken are added to a user'"'"'s electronic shopping cart to enable processing of a cashier-less transaction.

Claims

1. A method for identifying actions in a retail store, comprising:

(a) sampling a shopping environment using one or more sensors that include at least one camera capable of providing depth sensing to produce image data of a scene that shows a shopper in the retail store and tracking data related to one or more limbs of the shopper in connection to an item;

(b) receiving output of the sampling as feature inputs to one or more machine learning classifier models to derive one or more labels characterizing a behavior state of the shopper in connection with a state of the item; and

(c) wherein at least one processing entity associated with the retail store detects the state of the item to change from one as item taken to one as item returned, and sensor data from said one or more sensors used to produce one or more labels that indicate the item as having been returned to a wrong location in the retail store.

2. The method of claim 1, further comprising,

causing said at least one processing entity to send a message identifying the item being in the wrong location to enable a worker associated with the retail store to return the item to a location in the retail store that was assigned for the item.

3. The method of claim 1, wherein the items taken and not changed to be items returned are added to an electronic shopping cart associated with the shopper.

4. The method of claim 1, wherein the feature inputs to the machine learning classifier models are sensor output from the one or more sensors, or engineered features, or labels from another machine learning classifier.

5. The method of claim 1, wherein sampling the shopping environment includes using output of at least two different sensors disposed in the retail store and creating engineered feature data for use by the machine learning classifier models to derive labels that characterize tracking activity of shoppers and items in the retail store.

6. The method of claim 1, wherein one of said sensors is a camera and feature engineering is performed in processing output of the camera to identify coded features of the item in the retail store while a shopper is interacting with said item.

7. The method of claim 1, wherein a code is disposed on items in the retail store, the code is used in connection with tracking of one or more of said items when present in the retail store.

8. The method of claim 1, wherein the sampling of the shopping environment includes using output of at least two sensors and conducting feature engineering to derive a new feature used as an input to one or more of the machine learning classifier models.

9. The method of claim 1, wherein the sensors include two cameras positioned to capture images that at least partially overlap each other in the retail store to capture at least two different perspectives of said item.

10. The method of claim 1, wherein the state change or an additional state change characterized by the labels derived by the machine learning classifier models is associated to an event, or to an account, or to a user or shopper, or to guidance, or to feedback, or to a reward, or to an incentive, or to a sale, or to adding an item to an electronic shopping cart of the shopper, or to removing the item from the electronic shopping cart of the shopper.

11. A method of claim 1, wherein said sampling is repeated during a period of time during which the depth sensing produces multiple images and tracking data used by the classifier models to identify the behavior state of the shopper as including the item taken and then item returned by the shopper while in the retail store.

12. A method of identifying an event of a shopper taking an item in a store where the item taken is automatically identified, comprising:

(a) sampling a shopping environment using one or more sensors to include a reach in toward the item by a limb of the shopper;

(b) using a processing entity associated with the store to process an output of the sampling to identify the item taken by the shopper based on the reach in updating a state of the item that is labeled using processed features derived form the sampling,

(c) using a processing entity associated with the store to process the output of the sampling to identify the item put down by the shopper based on further sampling in (a) that includes the put down and processing in (b) for updating the state of the item,

(d) using a processing entity associated with the store to process the output of the sampling to identify the item put down by the shopper as being put in a wrong location in the store.

13. The method of claim 12, wherein after the item was determined to be put down in the wrong location in the store, a log is updated that identifies the state of the item to enable finding of the item and movement of the item to a correct location within the store.

14. The method of claim 12 wherein at least one sensor is a camera, and one or more items in the store have a code disposed thereon, the code being used in connection with tracking of said items in the store, wherein said tracking assists in identify said item as taken by the shopper and put down by the shopper or taken by the shopper for adding said item to an electronic shopping cart of the shopper.

15. The method of claim 12, further comprising,

executing one or more machine learning classifiers to process output from said one or more sensors, the outputs being input features to the one or more machine learning classifiers that produce labels characterizing said item or items as taken by the shopper or shoppers, or returned by the shopper or shoppers, or taken for addition to an online shopping chart used to charge said shopper or respective shopper for said item or items in a cashier-less transaction.

16. A method for identifying actions in a store, comprising:

capturing data in a shopping environment using one or more sensors including a camera to include a reach in toward an item by a limb of a shopper;

receiving output of the capturing as feature inputs to a machine learning classifier model to derive one or more labels characterizing a state of the item;

processing by at least one processing entity associated with the store to detect a change in the state of the item that indicates a take of the item from a location of the store based on said one or more labels used to identify the reach in as associated with the state of the item;

associating the take of the item to a user account of the shopper based on presence of a user device of the shopper at the store; and

charging the item to an electronic shopping cart for the user account upon confirming that the user device of the shopper has exited the store with the item.

17. The method of claim 16, wherein the confirming that the user device of the shopper has exited the store with the item includes using said captured data from at least said camera or one other camera associated with the store, wherein charging for said item is a cashier-less transaction.

18. The method of claim 16, wherein said machine learning classifier model derives additional labels to characterize additional items as taken, associating the additional items to the user account and charging said additional items to the electronic shopping cart upon confirming that the user device of the shopper has exited the store with said additional items, wherein confirming that the user device of the shopper has exited the store with the item and said additional items includes using said captured data from at least said camera.

19. The method of claim 16, wherein the feature inputs to the machine learning classifier are sensor output from the one or more sensors, or engineered features, or labels from another machine learning classifier, wherein the one or more sensors includes the camera and other cameras, or the camera and weight sensors, or the camera and motion sensors, or the camera and sound sensors, or the camera and microphones, or the camera and biometric sensors, or the camera and motion tracking sensors, or the camera and infrared sensors, or the camera and proximity detection sensors, or the camera and audio sensors, or the camera and a combination of one or more of said other cameras, weight sensors, sounds sensors, microphone sensors, biometric sensors, motion tracking sensors, infrared sensors, proximity detection sensors.

20. The method of claim 16, wherein capturing data from the shopping environment includes using output of at least two different sensors disposed in the store and creating engineered feature data for use by the machine learning classifier model to derive labels that characterize tracking activity of the shopper and items in the store.

21. The method of claim 16, wherein the sensors include the camera and at least one additional camera positioned to capture images that at least partially overlap each other in the store to capture at least two different perspectives of said item, or the sensors that include the camera and a plurality of other sensors in the store provide data that is analyzed by sensor fusion to identify information for making decisions regarding said take of the item, or other additional items by the shopper or take evens by additional shoppers in the store, such that takes of items by each of said additional shoppers in the store are added to their respective electronic shopping carts and charged to their respective accounts when said other shoppers are determined to have exited the store with said items.

22. The method of claim 16, wherein confirming that the user device of the shopper has exited the store is based on tracking the shopper exiting a region of the store, or is based on a label characterizing a scenario in the store associated with the likeliness that the shopper has exited the store, or is based on a state of a wireless signal, or is based on image data, or is based on a computed condition, or is based on a timed condition, or is based on passive sensors to enable the shopper to walk in and walk out without showing an identifier or logging into an application, or based on one or more trackers external to the store.

23. The method of claim 16, further including a second camera producing additional feature input to the machine learning classifier model, the second camera partially overlapping a scene captured by another camera in the store, the second camera being a depth camera or a traditional camera or an infrared camera or a camera having an array of sound sensors.

24. The method of claim 16, further including a second camera producing feature input into a second machine learning model, the second machine learning model and the first machine learning classifier model results being nested or cascaded or combined or used separately to generate a label to infer the actions of the shopper.

25. The method of claim 16, further including engineered feature data based on processing the output of the one or more sensors, the engineered feature data being combined as additional feature input to the machine learning classifier model used to generate the output label or used to engineer additional feature input to another machine learning model associated with performing a prediction to classify or infer the actions of the shopper.