Video analytics for retail business process monitoring

• US 9,158,975 B2
• Filed: 07/13/2007
• Issued: 10/13/2015
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Lipton, Alan J., Venetianer, Peter L., Haering, Niels, Brewer, Paul C., Yin, Weihong, Zhang, Zhong, Yu, Li, Hu, Yongtong, Myers, Gary W., Chosak, Andrew J., Cutting, Robert A., Scanlon, W. Andrew
• Primary Examiner(s): Survillo, Oleg
• Application Number:
  US11/826,324
• Publication Number:
  US 20080018738A1
• Time in Patent Office: 3,014 Days
• Field of Search: 348/143, 348/150
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19652 Systems using zones in a single scene defined for different treatment, e.g. outer zone gives pre-alarm, inner zone gives alarm
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system

Abstract

A system for video monitoring a retail business process includes a video analytics engine to process video obtained by a video camera and generate video primitives regarding the video, A user interface is used to define at least one activity of interest regarding an area being viewed, each activity of interest identifying at least one of a rule or a query regarding the area being viewed. An activity inference engine processes the generated video primitives based on each defined activity of interest to determine if an activity of interest occurred in the video.

Claims

1. A method for video monitoring a retail business process comprising:

obtaining video from a video camera;

processing, by one or more processors within one or more devices, the video obtained from the video camera;

generating, by one or more processors within one or more devices, video primitives regarding the video, wherein the video primitives comprise a high-value item stealing video primitive;

defining at least one activity of interest regarding an area being viewed, wherein an activity of interest identifies at least one of a user-defined rule or a user-defined query regarding the area being viewed, wherein at least one of the user-defined rule or the user-defined query comprises detection of high-value item stealing based on counting, using only the video, a number of times a person removes high-value items off a shelf;

wherein the number of times the person removes the high-value items off the shelf is counted using only the video by analyzing a motion pattern in the video to detect periodic motion based on an array of motion block information;

wherein the array of motion block information comprises motion blocks associated with a person removing high-value items off the shelf, wherein the motion blocks are generated based on one or more foreground masks and include a predetermined number of foreground pixels, wherein the predetermined number of foreground pixels is a user-defined parameter;

processing, by one or more processors within one or more devices, the generated video primitives based on at least one defined activity of interest to determine that an activity of interest occurred in the video based on determining that the number of times the person removes the high-value items off the shelf exceeds a user-defined threshold; and

generating an alert based on determining that the number of times the person removes the high-value items off the shelf exceeds the user-defined threshold.

2. The method of claim 1, wherein an array of motion block information associated with the person near the high-value item comprises positions and directions of motion blocks associated with the person near the high-value item.

3. A non-transitory computer-readable medium comprising software for video monitoring a retail business process, which software, when executed by a computer system, causes the computer system to perform operations comprising a method of:

processing video obtained by a video camera;

generating video primitives regarding the video, wherein the video primitives comprise a high-value item stealing video primitive;

defining at least one activity of interest regarding an area being viewed, wherein an activity of interest identifies at least one of a user-defined rule or a user-defined query regarding the area being viewed, wherein at least one of the user-defined rule or the user-defined query comprises detection of high-value item stealing based on counting, using only the video, a number of times a person removes high-value items off a shelf;

wherein the number of times the person removes the high-value items off the shelf is counted using only the video by analyzing a motion pattern in the video to detect periodic motion based on an array of motion block information;

wherein the array of motion block information comprises motion blocks associated with a person removing high-value items off the shelf, wherein the motion blocks are generated based on one or more foreground masks and include a predetermined number of foreground pixels, wherein the predetermined number of foreground pixels is a user-defined parameter;

processing the generated video primitives based on at least one defined activity of interest to determine that an activity of interest occurred in the video based on determining that the number of times the person removes the high-value items off the shelf exceeds a user-defined threshold; and

generating an alert based on determining that the number of times the person removes the high-value items off the shelf exceeds the user-defined threshold.

4. The non-transitory computer-readable medium of claim 3, wherein the array of motion block information associated with the person near the high-value item comprises at least one of up directions or down directions of motion blocks associated with the person near the high-value item.

5. An apparatus for video monitoring a retail business process comprising:

a video camera configured to obtain video of an area;

a video analytics engine configured to process the obtained video and generate video primitives regarding the video, wherein the video primitives comprise a high-value item stealing video primitive; and

an activity inference engine configured to process the generated video primitives based on at least one activity of interest regarding an area being viewed to determine if an activity of interest occurred in the video, wherein an activity of interest defines at least one of a rule or a query selectively identified by a user regarding the area being viewed, wherein at least one of the rule or the query comprises detection of high-value item stealing based on counting, using only the video, a number of times a person removes high-value items off a shelf, wherein the number of times the person removes the high-value items off the shelf is counted using only the video by analyzing a motion pattern in the video to detect periodic motion based on an array of motion block information;

wherein the array of motion block information comprises motion blocks associated with the person removing the high-value items off the shelf, wherein the motion blocks are generated based on one or more foreground masks and include a predetermined number of foreground pixels, wherein the predetermined number of foreground pixels is a user-defined parameter; and

an alert interface engine coupled to the activity inference engine configured to generate an alert based on determining that the number of times the person removes the high-value items off the shelf exceeds a user-defined threshold.

6. The apparatus as in claim 5, wherein the video analytics engine is resident within one of a chip, a chip set, or chips.

7. The apparatus as in claim 5, wherein the video analytics engine and the activity inference engine are resident within one of a chip, a chip set, or chips.

8. The apparatus as in claim 5, further including:

a plurality of video cameras which at least one obtains video of an associated area, wherein the video analytics engine processes the obtained video of at least one area and generates video primitives regarding the video and wherein the activity inference engine processes the generated video primitives based on at least one activity of interest regarding at least one area being viewed to determine if an activity of interest occurred in an associated video.

9. A system for video monitoring a retail business process comprising:

one or more processors within one or more devices configured to execute;

a video analytics engine configured to process video obtained by a video camera and to generate video primitives regarding the video, wherein the video primitives comprise a high-value item stealing video primitive;

a user interface configured to define at least one activity of interest regarding an area being viewed, wherein an activity of interest identifies at least one of a user-defined rule or a user-defined query regarding the area being viewed, wherein at least one of the user-defined rule or the user-defined query comprises detection of high-value item stealing based on counting, using only the video, a number of times a person removes high-value items off a shelf, wherein the number of times the person removes the high-value items off the shelf is counted using only the video by analyzing a motion pattern in the video to detect periodic motion based on an array of motion block information;

wherein the array of motion block information comprises motion blocks associated with the person removing the high-value items off the shelf, wherein the motion blocks are generated based on one or more foreground masks and include a predetermined number of foreground pixels, wherein the predetermined number of foreground pixels is a user-defined parameter;

an activity inference engine configured to process the generated video primitives based on at least one defined activity of interest and determine that an activity of interest occurred in the video based on determining that the number of times the person removes the high-value items off the shelf exceeds a user-defined threshold; and

an alert interface engine coupled to the activity inference engine configured to generate an alert based on determining that the number of times the person removes the high-value items off the shelf exceeds the user-defined threshold.

10. The system as in claim 9, wherein the video analytics engine and the activity inference engine are resident within a single device.

11. The system as in claim 9, wherein the video analytics engine and the activity inference engine are resident within the video camera.

12. The system as in claim 9, wherein the video analytics engine and the activity inference engine are resident within separate devices.

13. The system as in claim 9, wherein the video analytics engine operates in one of a real-time mode or an off-line mode.

14. The system as in claim 9, wherein the activity inference engine operates in one of a real-time mode or an off-line mode.

15. The system as in claim 9, wherein at least one defined activity of interest comprises at least one rule element and at least one combinator.

16. The system as in claim 9, further comprising:

a report generation engine coupled to the alert interface engine to generate a report based on one or more alerts received from the alert interface engine.

17. The system of claim 9, wherein the array of motion block information comprises x and y coordinates of one or more motion blocks and one or more timestamps.