IMAGE PROCESSING FOR TRACKING ACTIONS OF INDIVIDUALS

• US 20200134861A1
• Filed: 10/30/2018
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: NCR Voyix Corporation
• Original Assignee: NCR Corporation (NCR Voyix Corporation)
• Inventors: Zucker, Brent Vance, Lieberman, Adam Justin
• Granted Patent: US 11,055,874 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Cameras capture time-stamped images of predefined areas. Individuals and item are tracked in the images. A time-series set of images are processed to determine actions taken by the individuals with respect to the items or to determine relationships between the individuals to the items.

Claims

1. A method, comprising:

receiving a time-series set of images;

identifying from the time-series set of images an action taken by a tracked person with respect to a tracked item; and

providing a person identifier for the tracked person, an item identifier for the tracked item, and an action identifier for the action.

2. The method of claim 1, wherein receiving further includes receiving the time-series set of images as 8 sequentially taken image frames.

3. The method of claim 2, wherein identifying further creating a numerical matrix to hold the time-series set of images and cropping the time-series set of images to pixels associated with the tracked person within the numerical matrix.

4. The method of claim 3, wherein cropping further includes providing a reference to the numerical matrix as input to a trained neural network.

5. The method of claim 4, wherein providing further includes receiving the action identifier as output provided from the trained neural network.

6. The method of claim 2, wherein identifying further includes calculating a velocity and a direction of movement of the tracked item from the time-series set of images.

7. The method of claim 6, wherein calculating further includes applying rules to the velocity and the direction to resolve the action identifier.

8. The method of claim 7, wherein applying further includes applying the rules based on a known original location for the tracked item within a store.

9. The method of claim 1, wherein providing further includes providing the action identifier as an item-picked-up indication when the tracked person is identified from the time-series set of images as taking possession of the tracked item from a shelf of a store.

10. The method of claim 1, wherein providing includes providing the action identifier as an item-placed-back indication when the tracked person is identified from the time-series set of images as having placed the tracked item back on a shelf of a store.

11. A method, comprising:

tracking a person and an item representing in images;

processing a time-ordered set of the images to determine a relationship between the person and the item depicted in the time-ordered set; and

determining whether to add an item identifier for the item or remove the item identifier from a shopping cart maintained for a person identifier of the person based on the relationship.

12. The method of claim 11, wherein tracking further includes cropping first pixels associated with the person and second pixels associated with the item producing cropped person images and cropped item images from the images.

13. The method of claim 12, wherein processing further includes providing the cropped person images and the cropped item images to a trained neural network as input and receiving as output from the trained neural network a relationship identifier for the relationship.

14. The method of claim 11, wherein processing further includes applying rules to determine the relationship based on a computed a direction of movement for the item in the time-ordered set of images.

16. The method of claim 11, wherein processing further includes determining the relationship as an indication that the item was picked off a shelf within a store by the person.

17. The method of claim 11, wherein processing further includes determining the relationship as an indication that the item was placed back on a shelf within a store by the person.

18. The method of claim 11, wherein determining further includes providing the item identifier, the person identifier, and an add instruction or remove instruction to a transaction manager that manages the shopping cart for a transaction associated with the person within a store.

15. The method of claim 15, wherein applying further includes using a known shelf location for the item within a store and the direction of movement as at least one of the rules.

19. A system, comprising:

cameras configured to capture images within a store;

a server that includes a processor;

the processor configured to execute executable instructions from a non-transitory computer-readable storage medium as an action manager;

the action manager when executed on the processor configured to;

track an individual and an item identified in images;

determine an action taken by the individual with respect to the item from the images;

add an item identifier for the item to a shopping cart assigned to a person identifier for the person when the action indicates that the person picked the item off a shelf of a store; and

remove the item identifier for the item when present in the shopping cart when the action indicates that the person placed the item back on the shelf of the store.

20. The system of claim 19, wherein the action manager is further configured to determine the action by one of:

1) provide a time-ordered set of the images to a trained neural network and receive as output an action identifier for the action and

     2) determine the action identifier for the action based on a calculated velocity and direction of movement of the item within the time-ordered set of images relative to an original shelf location from the item within the store.