System and method for computer vision driven applications within an environment

• US 10,339,595 B2
• Filed: 05/09/2017
• Issued: 07/02/2019
• Est. Priority Date: 05/09/2016
• Status: Active Grant

• Current Assignee: Grabango Co.
• Original Assignee: Grabango Co.
• Inventors: Glaser, William, Van Osdol, Brian
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US15/590,467
• Publication Number:
  US 20170323376A1
• Time in Patent Office: 784 Days
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G07G 1/0036 Checkout procedures
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system and method for computer vision driven applications in an environment that can include collecting image data across an environment; maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process that includes: classifying objects, tracking object locations, detecting interaction events, instantiating object associations in the environmental object graph, and updating the environmental object graph by propagating change in at least one object instance across object associations; and inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state. The system and method can be applied to automatic checkout, inventory management, and/or other system integrations.

Claims

1. A method for expediting a checkout process comprising:

collecting image data across an environment;

maintaining an environmental object graph from the image data wherein the environmental object graph is a data representation of computer vision classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having object associations;

wherein maintaining the environmental object graph comprises at least one instance of;

in a first region captured in the image data, classifying a first object and at least a shopper object,

in the first region, detecting an interaction event between the first object and the shopper object, and

updating the environmental object graph whereby the first object is associated with the shopper object;

inspecting objects that are associated with the shopper object and thereby generating a checkout list.

2. The method of claim 1, further comprising, in a second region captured in the image data, detecting the shopper object and initiating a checkout process with the checkout list.

3. The method of claim 2, further comprising accessing an account associated with the shopper object;

and wherein initiating the checkout process comprises charging the checkout list to the account after entering the second region.

4. The method of claim 3, wherein initiating the checkout process comprises communicating the checkout list to a checkout processing system in the second region.

5. The method of claim 4, further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list.

6. The method of claim 5, further comprising biasing product input of the checkout processing system for items in the checkout list.

7. The method of claim 2, wherein the first object is associated with the shopper object with an initial confidence level;

and wherein generating the checkout list comprises adding objects to the checkout list with confidence levels satisfying a confidence threshold.

8. The method of claim 2, wherein the first region is captured in image data from a first camera and the second region is captured in image data from a second camera.

9. The method of claim 2, wherein the first region and the second region are captured in image data from one camera.

10. The method of claim 2, wherein the first object is associated with the shopper object as a possessed object with a first confidence level;

and wherein maintaining the environmental object graph in a second instance comprises;

in a third region captured in the image data, classifying a second object and the shopper object,

associating the second object as a possessed object with the shopper object with a second confidence level, and

updating the environmental object graph wherein the first confidence level is altered at least partially in response to the second confidence level.

11. The method of claim 1, further comprising accessing an account associated with the shopper object;

and in an application instance of the account, presenting the checkout list.

12. The method of claim 1, wherein maintaining the environmental object graph in a second instance after the first instance comprises:

in the first region, classifying the first object, and updating the environmental object graph and thereby removing the association of the first object with the shopper object in the first instance.

13. The method of claim 1, wherein maintaining the environmental object graph in a second instance comprises:

in a second region captured in the image data, classifying a second object and the shopper object, wherein the second object is a compound object with contained objects,

in the second region, detecting an interaction event between the first object and the shopper object, and

updating the environmental object graph whereby the second object and the contained objects of the second object are associated with the shopper object.

14. The method of claim 1, wherein detecting the interaction event between the first object and the shopper object comprises detecting proximity between the first object and the shopper object satisfying a proximity threshold.

15. The method of claim 1, wherein classifying the first object and the shopper object comprises applying computer-vision driven processes during classification, the computer-vision driven processes including at least image feature extraction and classification and an application of neural networks.

16. A method comprising:

collecting image data across an environment;

maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process comprising;

classifying objects and storing corresponding object instances in the environmental object graph,

tracking object locations and establishing an association of object instances in an object path,

detecting interaction events and, for event instances of a subset of detected interaction events, generating an object association of at least two object instances involved in the interaction event, and

updating the environmental object graph comprising propagating change in at least one object instance across object associations; and

inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state.

17. The method of claim 16, wherein collecting imaging data comprises collecting imaging data from multiple image capture devices distributed across an environment.

18. The method of claim 17, wherein collecting image data from multiple image capture devices distributed across an environment comprises collecting imaging data from a set of image capture devices that include at least two image capture configurations selected from:

an inventory storage capture configuration, an interaction capture configuration, an object identification capture configuration, and a movable capture configuration.

19. The method of claim 16, wherein detecting the interaction events comprises detecting proximity between a first object of a first classification and at least a second object of a second classification satisfying a proximity threshold.

20. The method of claim 16, wherein detecting the interaction events comprises:

for at least one interaction event, detecting an object proximity event, and, for at least a second interaction event, detecting an object transformation event.

21. The method of claim 20, wherein detecting the object transformation event can comprise detecting an object transformation selected from the set of:

an object appearance event, object disappearance event, and object classification mutation event.

22. The method of claim 16, wherein classifying an object can include identifying an object as one of a set of possible object types, wherein possible object types comprise at least a product object, a compound object, and a person object.

23. The method of claim 16, wherein maintaining the environmental object graph further comprises instantiating an object instance in the environmental object graph as probabilistically having a hierarchical association with at least one object.

24. The method of claim 16, wherein maintaining the environmental object graph further comprises instantiating at least two probabilistically possible states of an object instance.

25. The method of claim 16, wherein classifying objects comprises classifying an object through an initial classification model of a first set of candidate objects and classifying the object through at least a second expanded classification model of a second set of candidate objects if classification of the object through the initial classification model does not satisfy a classification condition.

26. The method of claim 16, wherein classifying objects comprises retrieving supplementary inputs selected from the set of:

purchase patterns, customer purchase history, and product pricing; and

classifying an object at least partially based on one of the supplementary inputs.

27. The method of claim 16, wherein collecting image data includes collecting image data at a checkout processing system;

and applying a product identifier generated by a point of sale input as supplemental input in updating the environmental object graph.

28. The method of claim 16, wherein inspecting object state and executing an action can comprise:

detecting a low confidence scenario in the environmental object graph;

communicating an environmental object graph calibration request specifying at least one region of the environment;

receiving calibration data input from the region, and updating the environmental object graph with the calibration data input.

29. The method of claim 16, wherein inspecting object state comprises generating a checkout list of product object instances associated with a shopper associated object instance.

30. The method of claim 29, wherein executing an action comprises:

accessing an account associated with the shopper associated object instance and charging the checkout list to the account when the shopper associated object instance is detected in a checkout region.

31. The method of claim 29, wherein executing an action comprises:

communicating the checkout list to a checkout processing system in proximity to the shopper associated object instance.

32. The method of claim 16, wherein executing an action comprises providing an inventory management system at least partially driven through the environmental object graph, wherein providing the inventory management system comprises:

reporting shelf time of products, reporting product counts, reporting product locations, and reporting container contents.

33. The method of claim 16, further comprising for a set of media content segments, determining a modeling confidence in the environmental object graph;

selecting a content representation based on the modeling confidence, the content representation selected from the set of options including at least a modeled data representation and a media representation; and

storing the content representation.

34. The method of claim 16, further comprising configuring an object rule with at least one condition and an action;

wherein inspecting object state and executing an action comprises detecting occurrence of the object rule and triggering the action of the object rule.

35. The method of claim 16, wherein classifying objects comprises applying computer-vision driven processes, the computer-vision driven processes including at least image feature extraction and classification and convolution neural networks.

36. A system comprising

an imaging system configured to collect image data within an environment;

a processing engine configured to maintain an environmental object graph from the image data wherein the environmental object graph is a data representation of classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having probabilistic object associations; and

the processing engine further configured to;

classify objects in the image data and store corresponding object instances in the environmental object graph,

track object locations and establish an association of object instances in an object path,

detect interaction events and, for event instances of a subset of detected interaction events, generate an object association of at least two object instances involved in the interaction event, and

update the environmental object graph and propagate change in at least one object instance across object associations.

37. The system of claim 36, wherein the imaging system comprises an image capture device in an inventory storage capture configuration, image capture device in an interaction capture configuration, an image capture device in an object identification capture configuration, and image capture device in a movable capture configuration.

38. The system of claim 36, wherein the processing engine is further configured to:

instantiate a first set of object instances in the environmental object graph with compound object modeling, and instantiate a second set of object instances in the environmental object graph with probabilistic multi-state modeling.

39. The system of claim 36, further comprising a virtual cart management system configured to inspect object instances associated with a shopper object instance in the environmental object graph and generate a checkout list of products.

40. The system of claim 39, the virtual cart management system further configured to access an account associated with the shopper object instance and charge the checkout list to the account when the shopper associated object instance is detected in a checkout region.

41. The system of claim 39, further comprising a checkout processing system wherein the checkout list is communicated to the checkout processing system.

42. The system of claim 36, further comprising an inventory management system that comprises an inspection interface to the environmental object graph.

43. The system of claim 36, further comprising a meta-media storage system that is configured to store a dynamic media content representation in coordination with the environmental object graph.

44. The system of claim 36, further comprising a calibration tool that is configured to provide calibration data input to the processing engine.