Busyness defection and notification method and system

• US 9,576,371 B2
• Filed: 02/19/2015
• Issued: 02/21/2017
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Mongeon, Michael C., Loce, Robert Paul, Shreve, Matthew Adam
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US14/625,960
• Publication Number:
  US 20150310312A1
• Time in Patent Office: 733 Days
• Field of Search: 382/103, 382/100, 382/159, 382/224, 382/181, 382/250, 382/232, 382/248, 382/190, 382/199, 382/195, 382/276, 382/293, 382/298, 382/299, 382/300, 709/203, 709/201, 709/219, 709/217, 709/223, 709/225, 345/518, 345/545, 345/530, 345/546, 345/555, 345/418, 345/428, 715/201, 715/200, 715/203, 715/221, 715/716, 715/723, 348/61, 348/143
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion

Abstract

This disclosure provides a video-based method and system for busyness detection and notification. Specifically, according to an exemplary embodiment, multiple overhead image capturing devices are used to acquire video including multiple non-overlapping ROIs (regions of interest) and the video is processed to count the number of people included within the ROIs. A busyness metric is calculated based on the number of people counted and notification of the busyness metric or changes in the busyness metric is communicated to appropriate personnel, e.g., a manager of a retail store.

Claims

1. A computer-implemented image based method of determining and communicating a busyness metric associated with a number of people within an area captured by one or more image capturing devices, the method comprising:

a) acquiring one or more images of the area from the one or more image capturing devices;

b) calculating a representation of the number of people within the area from the acquired one or more images of the area, the representation of the number of people calculated using a regression model trained with a training set of images, the regression model configured to receive an input feature vector associated with an acquired image and output a number of distinct people associated with the input feature vector associated with the acquired image;

c) calculating the busyness metric as a function of the representation of the number of people within the area, the busyness metric related to the calculated representation of people within the area during a sequence of acquired images representative of a predetermined time period; and

d) communicating the busyness metric to one or more of an electronic device associated with a user, a data storage device and a post processing device,

wherein step b) includes;

b1) dividing each of the one or more acquired images into a plurality of windows;

b2) extracting feature vectors from each of the plurality of windows;

b3) concatenating the extracted feature vectors to form a single feature vector representation of the number of people within the area covered by the plurality of windows; and

b4) inputting the single feature vector to the regression model to determine the number of people within the area covered by the plurality of windows.

2. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein step d) electronically notifies a person if an anomalous event has occurred, the anomalous event defined as one or more of (a) a rapid change in the number of people entering or exiting the area, and (b) the number of people within the area differing significantly from an expected number of people within the area for a time of day associated with the captured video.

3. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein the area is associated with one of a retail store counter, a customer service center, a private event, a public event, an airport, a train station, a bus station, and a government building.

4. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein step b) determines the number of people within an ROI (region of interest) associated with each of a plurality of image capturing devices and combines the determined number of people to calculate a total number of people within the area.

5. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein the calculated busyness metric is compared with an upper threshold value and a lower threshold value.

6. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein the calculated busyness metric is compared with one or more predefined reference busyness metric values to determine if the area is busy, normal or light.

7. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein the busyness metric is communicated to the electronic device via one of email, a cellular text message, and a cellular call.

8. The computer-implemented image based method of determining and communicating a busyness metric according to claim 1, wherein the calculated busyness metric is compared to a store resource profile.

9. An image processing system comprising:

a memory and a processor in communication with the processor, the memory and processor configured to receive digital images from one or more image capturing devices directed on an area, and execute instructions to perform an image based method of determining and communicating a busyness metric associated with a number of people within the area, the method comprising;

a) acquiring video of the area from the one or more of image capturing devices;

b) calculating a representation of the number of people within the area from the acquired one or more images of the area, the representation of the number of people calculated using a regression model trained with a training set of images, the regression model configured to receive an input feature vector associated with an acquired image and output a number of distinct people associated with the input feature vector associated with the acquired image;

c) calculating the busyness metric as a function of the representation of the number of people within the area, the busyness metric related to the calculated representation of people within the area during a sequence of acquired images representative of a predetermined time period; and

d) communicating the busyness metric to one or more of an electronic device associated with a user, a data storage device and a post processing device,

wherein step b) includes;

b1) dividing each of the one or more acquired images into a plurality of windows;

b2) extracting feature vectors from each of the plurality of windows;

b3) concatenating the extracted feature vectors to form a single feature vector representation of the number of people within the area covered by the plurality of windows; and

b4) inputting the single feature vector to the regression model to determine the number of people within the area covered by the plurality of windows.

10. The image processing system according to claim 9, wherein step d) electronically notifies a person if an anomalous event has occurred, the anomalous event defined as one or more of (a) a rapid change in the number of people entering or exiting the area, and (b) the number of people within the area differing significantly from an expected number of people within the area for a time of day associated with the captured video.

11. The image processing system according to claim 9, wherein the area is associated with one of a retail store, a customer service center, an industrial building, factory, a private event, a public event, a transportation setting including an airport, a train station, a bus station, and a government building.

12. The image processing system according to claim 9, wherein step b) determines the number of people within an ROI (region of interest) associated with each of a plurality of image capturing devices and combines the determined number of people to calculate a total number of people within the area.

13. The image processing system according to claim 9, wherein the calculated busyness metric is compared with an upper threshold value and a lower threshold value.

14. The image processing system according to claim 9, wherein the calculated busyness metric is compared with one or more predefined reference busyness metric values to determine if the area is busy, normal or light.

15. The image processing system according to claim 9, wherein the busyness metric is communicated to the electronic device via one of email, a cellular text message, cellular call, and webpage.

16. The image processing system according to claim 9, wherein the calculated busyness metric is compared to a store resource profile.

17. A computer-implemented image based method of determining and communicating a busyness metric associated with a number of people within an area captured by one or more image capturing devices, the method comprising:

a) acquiring one or more images of the area from the one or more image capturing devices;

b) calculating a representation of the number of people within the area from the acquired one or more images of the area, the representation of the number of people calculated using a regression model trained with a training set of images, the regression model configured to receive an input feature vector associated with an acquired image and output a number of distinct people associated with the input feature vector associated with the acquired image;

c) calculating the busyness metric as a function of the representation of the number of people within the area, the busyness metric related to the calculated representation of people within the area during a sequence of acquired images representative of a predetermined time period; and

d) communicating the busyness metric to one or more of an electronic device associated with a user, a data storage device and a post processing device,

wherein step b) comprises;

b1 ) performing background image subtraction on the one or more acquired images to generate one or more respective foreground images associated with the one or more acquired images;

b2 ) counting foreground pixels, blobs, and/or edge pixels associated with the one or more foreground images; and

b3 ) inputting the counted foreground pixels, blobs, and/or edge pixels, represented as a feature vector, to the regression model to determine the number of people within the area.

18. An image processing system comprising:

a memory and a processor in communication with the processor, the memory and processor configured to receive digital images from one or more image capturing devices directed on an area, and execute instructions to perform an image based method of determining and communicating a busyness metric associated with a number of people within the area, the method comprising;

a) acquiring video of the area from the one or more of image capturing devices;

b) calculating a representation of the number of people within the area from the acquired one or more images of the area, the representation of the number of people calculated using a regression model trained with a training set of images, the regression model configured to receive an input feature vector associated with an acquired image and output a number of distinct people associated with the input feature vector associated with the acquired image;

c) calculating the busyness metric as a function of the representation of the number of people within the area, the busyness metric related to the calculated representation of people within the area during a sequence of acquired images representative of a predetermined time period; and

d) communicating the busyness metric to one or more of an electronic device associated with a user, a data storage device and a post processing device,

wherein step b) comprises;

b1 ) performing background image subtraction on the one or more acquired images to generate one or more respective foreground images associated with the one or more acquired images;

b2 ) counting foreground pixels, blobs, and/or edge pixels associated with the one or more foreground images; and

b3 ) inputting the counted foreground pixels, blobs, and/or edge pixels, represented as a feature vector, to the regression model to determine the number of people within the area.