System and method for monitoring a retail environment using video content analysis with depth sensing

• US 9,740,937 B2
• Filed: 01/17/2013
• Issued: 08/22/2017
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Zhang, Zhong, Myers, Gary W., Venetianer, Peter L.
• Primary Examiner(s): Cese, Kenny
• Application Number:
  US13/744,251
• Publication Number:
  US 20130182114A1
• Time in Patent Office: 1,678 Days
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0046 Arrangements of imaging apparatus in a room, e.g. room provided with shielding or for improved access to apparatus
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1072 measuring distances on the body, e.g. measuring length, height or thickness A61B5/1076 takes precedence
• CPC Class Codes: A61B 5/1073 Measuring volume, e.g. of limbs
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 : Event detection
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/1365 : Matching; Classification
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 21/043 : detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0476 : Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A method and system for monitoring a retail environment by performing video content analysis based on two-dimensional image data and depth data are disclosed. Accuracy in customer actions to provide assistance, change marketing behavior, safety and theft, for example, is increase by analyzing video containing two-dimensional image data and associated depth data. Height data may be obtained from depth data to assist in object detection, object classification (e.g., detection a customer or inventory) and/or event detection.

Claims

1. A method of monitoring a retail environment comprising:

taking a video at a store with a video sensor, the video comprising a plurality of frames, each frame including two-dimensional (2D) image data;

for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from the video sensor to features represented by the 2D image data;

analyzing the 2D image data without analyzing the depth data to detect one or more objects in the video;

using the depth data to classify the one or more detected objects in the store depicted in the video, classification of the one or more detected objects comprising at least one of person classification and inventory classification and being based on a volume of the one or more detected objects; and

detecting an event of the classified one or more objects,

wherein the volume is determined by using the depth data along with the 2D image data to determine a plurality of convex hull slices on different Z-planes, and by summing areas of the plurality of convex hull slices.

2. The method of claim 1,

wherein the classification of the one or more objects comprises person classification, and

wherein the detecting an event of the classified one or more objects comprises using the depth data to detect a detected person is lying down in the store.

3. The method of claim 2, wherein the detection of a person lying down comprises identifying one or more parts of the detected person and comparing height information of the one or more parts of the detected person with a height of the floor of the store.

4. The method of claim 1, wherein the detecting an event of the classified one or more objects comprises using the depth data associated with the 2D image data to detect a spill of a substance on a floor of the store.

5. The method of claim 4, wherein the detection of a spill comprises obtaining height information of a first object and determining that the height information indicates the first object is within a first distance of the height of the floor.

6. The method of claim 1,

wherein analyzing the 2D image data and using the depth data to classify the one or more objects detects and classifies plural objects, each as a respective person, and

wherein the detecting of the event comprises counting a number of people.

7. The method of claim 6, further comprising monitoring at least one of a height and a distance from a depth sensor providing the depth data of an upper portion of each object classified person.

8. The method of claim 7, further comprising:

for each of the objects classified as a person, associating a depth value with a portion of the person most distant from the floor of the store.

9. The method of claim 7, further comprising determining a height value of each object classified as a person, and using the height value to track each object classified as a person.

10. The method of claim 9, wherein the detecting of the event comprises counting a number of people entering the store.

11. The method of claim 9, wherein the detecting of the event comprises counting a number of people in a line of the store.

12. The method of claim 1, wherein using the depth data to classify the one or more objects comprises obtaining height data of at least some of the detected one or more objects from corresponding depth data of the at least some of the detected one or more objects.

13. The method of claim 12, wherein the height data is used to classify a person as an adult.

14. The method of claim 13, wherein the height data is used to classify a person as a child.

15. The method of claim 12, wherein the height data is used to classify an object as an object located in a shopping cart.

16. The method of claim 15, further comprising classifying a type of the object located in the shopping cart.

17. The method of claim 12, wherein the height data is used to classify an object as a shopping cart.

18. The method of claim 17, further comprising tracking shopping cart movement within the store.

19. The method of claim 1, wherein a first object is classified as a person, and detecting an event of the person comprises detecting a reaching of the person.

20. The method of claim 19, wherein detecting the reaching of the person comprises detecting an arm height of the person from the depth data.

21. The method of claim 20, wherein detecting the reaching of the person comprises comparing the detected arm height of the person with a known height of a background feature represented by the image data.

22. The method of claim 1, wherein a first object is classified as a person, and detecting an event of the person comprises detecting a time the person dwells within a certain location within the store.

23. The method of claim 22, further comprising generating an alert in response to detecting that the person has dwelled within a certain location greater than a predetermined time.

24. The method of claim 22, further comprising analyzing dwell times of multiple customers over plural weeks to determine a pattern of dwell times of customers of the store.

25. The method of claim 24, further comprising using the pattern of dwell times to determine employee allocation within the store.

26. The method of claim 24, further comprising using the pattern of dwell times to adjust merchandising within the store.

27. The method of claim 1, wherein a first object is classified as inventory, and detecting an event of the inventory comprises detecting that the inventory is low or absent.

28. The method of claim 27, wherein the depth data is used to determine a depth of the inventory on a shelf, within a refrigerator, or within a freezer in the store, and wherein the depth data is used to detect that the inventory is low or absent.

29. The method of claim 28, further comprising generating an alert in response to determining that the inventory is low or absent to notify at least one of store management or inventory suppliers of the need to supplement the inventory.

30. The method of claim 1, wherein a first object is classified as inventory, and detecting an event of the inventory comprises detecting that the inventory is delivered at a delivery location of the store.

31. The method of claim 30, wherein the depth data is used to detect appropriate delivery of inventory to the store.

32. The method of claim 1, wherein a first object is classified as inventory, and detecting an event of the inventory comprises tracking the inventory to detect if the inventory is removed without delivery.