Computer vision system and method for automatic checkout

• US 10,614,514 B2
• Filed: 05/17/2019
• Issued: 04/07/2020
• Est. Priority Date: 05/09/2016
• Status: Active Grant

• Current Assignee: Grabango Co.
• Original Assignee: Grabango Co.
• Inventors: Glaser, William, Van Osdol, Brian
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/416,009
• Publication Number:
  US 20190378205A1
• Time in Patent Office: 326 Days
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system and method for computer vision driven applications in an environment that can include collecting image data across an environment; maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process that includes: classifying objects, tracking object locations, detecting interaction events, instantiating object associations in the environmental object graph, and updating the environmental object graph by propagating change in at least one object instance across object associations; and inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state. The system and method can be applied to automatic checkout, inventory management, and/or other system integrations.

Claims

1. A method for a checkout process comprising:

collecting image data across an environment;

maintaining an environmental object graph from the image data wherein the environmental object graph is a data representation of computer vision classified objects across the environment, the environmental object graph comprising at least a subset of objects having object associations;

wherein maintaining the environmental object graph comprises at least one instance of;

in a first region captured in the image data, classifying a product object and at least a shopper object,

in the first region, detecting an interaction event between the product object and the shopper object, and

updating the environmental object graph whereby the product object is associated with the shopper object;

generating a checkout list from product objects associated with the shopper object in the environmental object graph.

2. The method of claim 1, further comprising, in a second region captured in the image data, detecting the shopper object and initiating a checkout process with the checkout list.

3. The method of claim 2, further comprising accessing an account associated with the shopper object;

and wherein initiating the checkout process comprises charging the checkout list to the account after entering the second region.

4. The method of claim 3, wherein initiating the checkout process comprises communicating the checkout list to a checkout processing system in the second region.

5. The method of claim 4, further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list.

6. The method of claim 1, further comprising accessing an account associated with the shopper object;

and in an application instance of the account, presenting the checkout list.

7. The method of claim 1, further comprising communicating the checkout list to a checkout processing system.

8. The method of claim 7, further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list.

9. The method of claim 1, classifying the product object comprises classifying a stock keeping unit identifier of the product object.

10. The method of claim 1, wherein classifying the product object comprises retrieving supplementary inputs selected from the set of:

purchase patterns, customer purchase history, and product pricing; and

classifying the product object at least partially based on one of the supplementary inputs.

11. A method comprising:

collecting image data across an environment;

maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process comprising;

classifying objects, which comprises of classifying a first subset of objects as product objects and classifying a second subset of objects as shopper objects,

storing object instances corresponding to the objects in the environmental object graph,

detecting interaction events, and

for a subset of detected interaction events between product object and a shopper object, updating the environmental object graph by associating the product object with the shopper object; and

inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state.

12. The method of claim 11, wherein detecting the interaction events comprises:

for at least one interaction event, detecting an object proximity event, and, for at least a second interaction event, detecting an object transformation event.

13. The method of claim 12, wherein detecting the object transformation event can comprise detecting an object transformation selected from the set of:

an object appearance event, object disappearance event, and object classification mutation event.

14. The method of claim 11, wherein classifying an object can include identifying an object as one of a set of possible object types, wherein possible object types comprise of at least a product object type, a compound object type, and a person object type.

15. The method of claim 11, wherein maintaining the environmental object graph further comprises instantiating an object instance in the environmental object graph as having a hierarchical association with at least one object.

16. The method of claim 11, wherein classifying objects comprises retrieving supplementary inputs selected from the set of:

purchase patterns, customer purchase history, and product pricing; and

classifying an object at least partially based on one of the supplementary inputs.

17. The method of claim 11, wherein collecting image data includes collecting image data at a checkout processing system;

and applying a product identifier generated by a point of sale input as supplemental input in updating the environmental object graph.

18. The method of claim 11, wherein inspecting object state and executing an action can comprise:

detecting a low confidence scenario in the environmental object graph;

communicating an environmental object graph calibration request specifying at least one region of the environment;

receiving calibration data input from the region, and updating the environmental object graph with the calibration data input.

19. The method of claim 11, wherein executing an action comprises providing an inventory management system at least partially driven through the environmental object graph.

20. The method of claim 19, wherein providing the inventory management system comprises:

reporting shelf time of products, reporting product counts, reporting product locations, and reporting container contents.

21. The method of claim 11, wherein classifying objects comprises applying computer-vision driven processes, the computer-vision driven processes including at least image feature extraction and classification and convolution neural networks.

22. The method of claim 11, classifying the product object comprises classifying a stock keeping unit identifier of the product object.

23. The method of claim 11, wherein inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state comprises of generating a checkout list of product objects associated with the shopper object.

24. The method of claim 23, further comprising initiating a checkout process with the checkout list.

25. The method of claim 24, further comprising accessing an account associated with the shopper object;

and wherein initiating the checkout process comprises charging the checkout list to the account.

26. The method of claim 24, wherein initiating the checkout process comprises communicating the checkout list to a checkout processing system in proximity to the shopper object.

27. The method of claim 26, further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list.

28. The method of claim 23, further comprising accessing an account associated with the shopper object;

and in an application instance of the account, presenting the checkout list.

29. A system comprising:

an imaging system comprised of a set of imaging devices to collect image data within an environment;

a processing engine comprised of at least one processor and machine instructions that when executed by the at least one processor cause the processing engine to;

maintain an environmental object graph from the image data wherein the environmental object graph is a data representation of classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having object associations,

classify objects in the image data including classifying a product objects and classifying shopper objects,

store object instances that correspond to the objects in the environmental object graph,

detect interaction events, and

for a subset of detected interaction events between product object and a shopper object, update the environmental object graph by associating the product object with the shopper object.

30. The system of claim 29, wherein the processing engine is further configured to:

instantiate a first set of object instances in the environmental object graph with compound object modeling, and instantiate a second set of object instances in the environmental object graph with multi-state modeling.

31. The system of claim 29, further comprising a virtual cart management system configured to inspect object instances associated with a shopper object instance in the environmental object graph and generate a checkout list of products.

32. The system of claim 31, the virtual cart management system further configured to access an account associated with the shopper object instance and charge the checkout list to the account when the shopper associated object instance is detected in a checkout region.

33. The system of claim 31, further comprising a checkout processing system wherein the checkout list is communicated to the checkout processing system.

34. The system of claim 31, further comprising a shopper application that presents the checkout list.

35. The system of claim 29, further comprising a calibration tool that is configured to provide calibration data input to the processing engine.