Methods and systems for goods received gesture recognition

• US 9,536,153 B2
• Filed: 08/05/2013
• Issued: 01/03/2017
• Est. Priority Date: 08/29/2012
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Shreve, Matthew Adam, Mongeon, Michael C., Loce, Robert Paul
• Primary Examiner(s): Teitelbaum, Michael
• Application Number:
  US13/959,195
• Publication Number:
  US 20150035975A1
• Time in Patent Office: 1,247 Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 20/30 characterised by the use of specific devices or networks
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Disclosed are methods and systems to automatically detect and optionally classify when a recipient receives goods in surveillance video. According to one exemplary embodiment, a computer implemented method determines a customer has received goods which are associated with a retail environment.

Claims

1. A computer implemented method of determining a customer has received one or more goods which are associated with a retail environment, the method comprising:

a) acquiring a surveillance video sequence using at least one video camera, the video camera oriented to include a field of view associated with a goods receiving area of the retail environment where the one or more goods are to be received by the customer;

b) processing the video sequence to detect motion representative of the one or more goods being placed in the goods receiving area by a retail environment employee;

c) processing the video sequence to track within the goods receiving area the one or more goods detected in step b);

d) processing the video sequence to detect the customer associated with receiving the one or more goods detected in step b);

e) processing the video sequence to determine the customer detected in step d) has received the one or more goods detected in step b);

wherein a Goods ROI (Region of Interest) is associated with the goods receiving area and a Customer ROI is associated with the goods receiving area, the Goods ROI including an area where the one or more goods are placed by the retail environment employee, the Customer ROI including an area where the customer receives the one or more goods, and the Goods ROI and Customer ROI overlap and include areas exclusive to the other ROI.

2. The computer implemented method according to claim 1, step b) comprising:

processing the video sequence using background substraction and thresholding to segment a foreground from a background to create a binary image, the foreground associated with pixel regions estimated to include the motion representative of the one or more goods being placed in the goods receiving area by the retail environment employee.

3. The computer implemented method according to claim 2, wherein one of a dynamic background model, a fixed background model and a background model generated using a mixture model is used to perform the background substraction.

4. The computer implemented method of claim 1, wherein

step b) processes the video sequence to detect motion representative of the one or more goods being placed by the retail environment employee within the Goods ROI;

step c) processes the video sequence to track within the Goods ROI the one or more goods detected in step b); and

step d) processes the video sequence to detect the customer within the Customer ROI associated with the one or more goods detected in step b).

5. The computer implemented method according to claim 1, wherein step b) includes:

processing the video sequence to detect blobs of pixels representative of one or more goods being placed in the goods receiving area by the retail environment employee; and

applying heuristic rules to the detected blobs to determine if a characteristic of the detected blob is associated with an object characteristic of the one or more goods received by the customer.

6. The computer implemented method according to claim 1, wherein step d) processes the video sequence to detect the customer associated with the one or more goods detected in step b) by determining customer movement towards the one or more goods tracked in step c).

7. The computer implemented method according to claim 1, wherein step e) determines the customer has received the one or more goods detected in step b) when the one or more goods is no longer detected in the goods receiving area and/or no longer being tracked in the goods receiving area.

8. The computer implemented method according to claim 1, wherein motion is estimated using an optical flow method including one of a Horn-Schunck method, a Lucas-Kanade method and a block flow method.

9. The computer implemented method according to claim 1, further comprising:

f) determining a Total Experience Time of the customer based in part on the step e) determination the customer has received the one or more goods.

10. An image capturing system comprising:

an image capturing device; and

an image processor operatively associated with the image capturing device, the image processor configured to perform a method of detecting a customer receiving one or more goods associated with a retail environment, the method comprising;

a) acquiring a surveillance video sequence using at least one video camera, the video camera oriented to include a field of view associated with a goods receiving area of the retail environment where the one or more goods are to be received by the customer;

b) processing the video sequence to detect motion representative of the one or more goods being placed in the goods receiving area by a retail environment employee;

c) processing the video sequence to track within the goods receiving area the one or more goods detected in step b);

d) processing the video sequence to detect the customer associated with receiving the one or more goods detected in step b);

e) processing the video sequence to determine the customer detected in step d) has received the one or more goods detected in step b);

wherein a Goods ROI (Region of Interest) is associated with the goods receiving area and a Customer ROI is associated with the goods receiving area, the Goods ROI including an area where the one or more goods are placed by the retail environment employee, the Customer ROI including an area where the customer receives the one or more goods, and the Goods ROI and Customer ROI overlap and include areas exclusive to the other ROI.

11. The image capturing system according to claim 10, step b) comprising:

processing the video sequence using background substraction and thresholding to segment a foreground from a background to create a binary image, the foreground associated with pixel regions estimated to include the motion representative of the one or more goods being placed in the goods receiving area by the retail environment employee.

12. The image capturing system according to claim 11, wherein one of a dynamic background model, a fixed background model and a background model generated using a mixture model is used to perform the background substraction.

15. The image capturing system according to claim 10, wherein step b) includes:

processing the video sequence to detect blobs of pixels representative of one or more goods being placed in the goods receiving area by the retail environment employee; and

applying heuristic rules to the detected blobs to determine if a characteristic of the detected blob is associated with an object characteristic of the one or more goods received by the customer.

16. The image capturing system according to claim 10, wherein step d) processes the video sequence to identify the customer associated with the one or more goods detected in step b) by determining customer movement towards the one or more goods tracked in step c).

17. The image capturing system according to claim 10, wherein step e) determines the customer has received the one or more goods detected in step b) when the one or more goods is no longer detected in the goods receiving area and/or no longer being tracked in the goods receiving area.

18. The image capturing system according to claim 10, wherein motion is estimated using an optical flow method including one of a Horn-Schunck method, a Lucas-Kanade method and a block flow method.

19. The image capturing system according to claim 10, the method further comprising:

f) determining a Total Experience Time of the customer based in part on the step e) determination the customer has received the one or more goods.

13. The image capturing system according to clam 10, wherein a Goods ROI (Region of Interest) is associated with the goods receiving area and a customer ROI is associated with the goods receiving area, the Goods ROI including an area where the one or more goods are placed by the retail environment employee and the Customer ROI including an area where the customer receives the one or more goods,

step b) processes the video sequence to detect motion representative of the one or more goods being placed by the provider within the Goods ROI;

step c) processes the video sequence to track within the Goods ROI the one or more goods detected in step b); and

step d) processes the video sequence to detect the customer within the Customer ROI associated with the one or more goods detected in step b).

14. The image capturing system according to claim 13, wherein the Goods ROI and Customer ROI overlap and include areas exclusive to the other ROI.

20. A computer implemented method of detecting a recipient has received one or more goods, the method comprising:

aa) acquiring a surveillance video using at least one video camera, the video camera oriented to include a field of view associated with a goods receiving area where the one or more goods are to be received by the recipient;

bb) processing the surveillance video to detect the one or more goods being placed in the goods receiving area by a retail environment employee;

cc) processing the surveillance video to track within the goods receiving area the one or more goods detected in step bb);

dd) processing the surveillance video to detect the recipient associated with receiving the one or more goods detected in step bb); and

ee) processing the surveillance video to determine the recipient detected in step dd) has received the one or more goods detected in step bb);

wherein a Goods ROI (Region of Interest) is associated with the goods receiving area and a Customer ROI is associated with the goods receiving area, the Goods ROI including an area where the one or more goods are placed by the retail environment employee, the Customer ROI including an area where the customer receives the one or more goods, and the Goods ROI and Customer ROI overlap and include areas exclusive to the other ROI.

21. The computer implemented method according to claim 20, wherein a Goods ROI (Region of Interest) is associated with the goods receiving area and a Recipient ROI is associated with the goods receiving area, the Goods ROI including an area where the one or more goods are placed by the retail environment employee and the Recipient ROI including an area where the customer receives the one or more goods,

step bb) processes the video sequence to detect motion representative of the one or more goods being placed by the retail environment employee within the Goods ROI;

step cc) processes the video sequence to track within the Goods ROI the one or more goods detected in step b); and

step dd) processes the video sequence to detect the recipient within the Recipient ROI associated with the one or more goods detected in step b).

22. The computer implemented method according to claim 21, wherein the Goods ROI and Recipient ROI overlap and include areas exclusive to the other ROI.

23. The computer implemented method according to claim 20, wherein step ee) determines the recipient has received the one or more goods detected in step bb) when the one or more goods is no longer detected in the goods receiving area and/or no longer being tracked in the goods receiving area.

24. The computer implemented method according to claim 20, wherein step bb) includes:

processing the video sequence to detect blobs of pixels representative of one or more goods being placed in the goods receiving area by the retail environment employee; and

applying heuristic rules to the detected blobs to determine if a characteristic of the detected blob is associated with an object characteristic of the one or more goods received by the customer.