Machine learning methods and system for tracking label coded items in a retail store for cashier-less transactions

• US 10,573,134 B1
• Filed: 09/26/2018
• Issued: 02/25/2020
• Est. Priority Date: 07/25/2015
• Status: Active Grant

• Current Assignee: Albert S. Penilla, Gary M. Zalewski
• Original Assignee: Albert S. Penilla, Gary M. Zalewski
• Inventors: Zalewski, Gary M., Penilla, Albert S.
• Primary Examiner(s): Chen, Alan
• Application Number:
  US16/143,449
• Time in Patent Office: 517 Days
• Field of Search: None
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/327 Short range or proximity payments by means of M-devices
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G07G 1/0072 with means for detecting the weight of the article of which the code is read, for the verification of the registration weighing combined with recording means, e.g. in weighing scale, G01G19/415

Abstract

Devices, systems, and method are provided for operating a store. One method includes sampling a shopping environment using one or more sensors to produce features relating to a state of activity of the store. The method further includes processing output of at least one sensor through a feature extractor to extract one or more additional features relating to the state of activity of the retail store. Then, processing at least part of the produced features and at least part of the extracted additional features using a processing entity associated with the store and a machine learning model to generate one or more labels relating to the state of activity of the retail store. In further embodiments, items in the store are physically taken into possession and automatically added to a user'"'"'s electronic shopping cart to enable processing of a cashier-less transaction.

Claims

1. A method for identifying actions in a retail store, comprising:

sampling a shopping environment by receiving by one or more processing entities associated with the store, output of one or more sensors disposed in the retail store, each sensor capable of producing said output as raw data reflecting an exposure of said each sensor to a scenario occurring within a sensing range of said each sensor in the retail store;

processing, by said one or more processing entities associated with the retail store, at least one sensor output to generate at least one extracted feature;

processing said at least one extracted feature, by said one or more processing entities associated with the retail store together with output from said one or more sensors disposed in the retail store, to produce feature input to a machine learning model to generate a label characterizing a state of the scenario occurring in the retail store, the state identifying interactions by a shopper as a take of an item from a shelf, the interactions of the shopper include a reach toward the shelf to perform the take of the item, the state of the scenario includes identification data of the item taken from the shelf.

2. The method of claim 1, wherein the scenario defines activity captured by said one or more sensors as said raw data in relation to the shopper in the retail store and said interactions of the shopper while in the retail store with at least said item of the retail store that is for sale, said activity being identified or tracked to enable said one or more processing entities to produce said feature input to the machine learning model that characterizes a state of the item in relation to the shopper in the retail store, the state of the item including said take of the item from the shelf.

3. The method of claim 1, wherein the one or more sensors disposed in the retail store include at least two cameras that are at least partially directionally overlapping each other such that capture of image data from each said cameras partially overlap a same area of said shopping environment.

4. The method of claim 1, wherein the scenario defines activity occurring in the retail store in relation between the item in the retail store and the shopper in the retail store, and associated one or more of said sensors disposed in the retail store.

5. The method of claim 1,

wherein output from said one or more sensors is used to produce said at least one extracted feature, and said output from said one or more sensors is further used to produce an additional extracted feature,

wherein each of said at least one extracted feature and said additional extracted feature are provided as inputs to the machine learning model.

6. The method of claim 5, wherein the at least one extracted feature identifies part of the raw data produced by said at least one or more sensors and said additional extracted feature identifies another part of the raw data when the machine learning model produces the labels characterizing the scenario occurring in the retail store.

7. The method of claim 3, wherein the extracted feature and additional extracted features are extracted over time from camera image data representing the raw data and the machine learning model produces the label and additional labels over time, said labels relating to state of items of the store, or related to one or more shoppers taking an item into their possession, or related to interest by the shopper in the item, or related to the shopper having returned the item to a shelf or location, or related to the shopper returning the item to a wrong place in the retail store, or related to an identity of the shopper, or related to detection of the shopper examining an alternative brand previously purchased by the shopper, or related to the shopper'"'"'s churn risk for an item in which the shopper is interacting.

8. The method of claim 1, further comprising,

providing a weight sensor coupled or interfaced with items of the retail store, an output of the weight sensor further being added as a feature input to machine learning model to generate an additional label characterizing the state of the scenario occurring in the retail store, wherein the item being taken is an item physically taken into possession by the shopper that causes the output of the weight sensor to be reduced.

9. The method of claim 1, further comprising,

the item taken is physically taken into possession from the retail store, and a processing entity associated with the retail store detects the item that has been taken by the shopper, and the item being added to an electronic shopping cart associated to an account linked to the shopper.

10. The method of claim 1, further comprising,

determining that the shopper returns item while in the retail store, a processing entity associated with the store detects the item that has been returned by the shopper, and the item being automatically removed from an electronic shopping cart associated to an account linked to the shopper.

11. The method of claim 1, further comprising,

determining that the shopper releases the item possessed by the shopper, a processing entity associated with the retail store detects the item that has been released by the shopper as now being in a wrong location in the retail store, the item being removed from an electronic shopping cart associated to an account linked to the shopper.

12. The method of claim 1, wherein the extracted feature is an identifier (ID) that relates to a coded item of the retail store, the coded item having a code that is part of a label associated with the item or part of a packaging of the item.

13. The method of claim 1, wherein the extracted feature is a detected object and the detected object is detected from image data of one of said sensors, wherein one of said sensors being a camera that produces images as said raw data, and wherein said detected object being a person or shopper or the item of the retail store.

14. The method of claim 1, wherein an identity of the shopper is tracked over time while the shopper is inside the retail store, or tracked over time while said shopper is inside a region of the retail store, or, where the identity of the shopper is periodically re-identified in the store.

15. The method of claim 1, wherein the shopper is identified and tracked while in the retail store, and further comprising,

enabling removal of a history of said shopper being identified or tracked while in the retail store by filtering out said history, wherein said filtering out said history removes learning assumptions for privacy control by the shopper in relation to one or more items in the store.

16. A method for identifying actions in a store, comprising:

sampling a shopping environment using one or more sensors; and

using a processing entity associated with the store for processing features related to the sampled shopping environment, the features are output from the one or more sensors using one or more machine learning classifiers and are further processed to derive one or more labels that classify activity of a shopper as the shopper moves about the store;

wherein at least one processing entity associated with the store detects from said activity that an item of the store has been taken by the shopper, the taken item being added to an electronic shopping cart associated to an account linked to the shopper detected to be interacting with the item;

wherein at least one sensor is a camera producing output including image data capturing one or more codes of items present in the store, said image data processed by said at least one processing entity to extract an ID associated with said item, wherein said item being one by which the shopper is identified to be interacting with and in response to detecting the shopper reaching toward a shelf to perform the take of the item, an inference is made reflecting a state of a shopping scenario, the inference made characterizes the shopping scenario involving the shopper, the taking of the item, and the item taken from the shelf, wherein the item determined to be taken is added to the electronic shopping cart.

17. The method of claim 16, wherein the one or more sensors are of two or more different types, wherein one type is the camera, wherein other types include a depth camera, or a weight sensor, or a microphone, or an infrared sensor, or a proximity sensor, or a combination thereof, and wherein sensor fusion is processed to improve said determination of said take of the item by the shopper or improve tracking of said shopper and limbs of the shopper as the shopper moves about the store.

18. The method of claim 16 wherein the machine learning classifier produces output related to tracking shopper activities in the store, at least one processing entity using the tracked shopper activities of the shopper to build a data profile associated with the shopper, said profile used for recommending items to said shopper,

wherein the tracked activities include observations that the shopper is determined to have toward items in the store, or the interactions the shopper has to the item while in the store, or to a gaze of the shopper while in the store, or a time spent in connection with the item or another item or a group of items, or a level of interest the shopper is determined to have toward an item, or a level of interest to an item held by the shopper, or a level of interest the shopper is determined to have toward an item returned by the shopper.

19. The method of claim 16 wherein the sensors include two cameras positioned to produce image data that at least partially overlaps in a region of the store, wherein the image data is from at least two different perspectives.

20. The method of claim 16, wherein the label output from the machine learning classifier reflects additional shopping activity of the shopper and a change in label is coupled to an event, or an account, or a user, or to guidance, or to feedback, or to a reward, or to an incentive, or to a sale, or to adding an item to the electronic shopping cart of the shopper, or to removing an item from the electronic shopping cart of the shopper, or to a data profile associated with the shopper that is used for recommending items to said shopper, or the data profile associated with the shopper that is used for determining a custom price for an item available for purchase by the shopper.