System and method for activity monitoring using video data

• US 10,432,896 B2
• Filed: 06/01/2015
• Issued: 10/01/2019
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Placemeter, Inc. (Arlo Technologies, Inc.)
• Original Assignee: Placemeter, Inc. (Arlo Technologies, Inc.)
• Inventors: Winter, Alexandre, Haldimann, Niklaus, Thi, Tuan Hue
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US14/727,321
• Publication Number:
  US 20150350608A1
• Time in Patent Office: 1,583 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Embodiments of a method and system described herein enable capture of video data streams from multiple, different devices and the processing of the video data streams. The video data streams are merged such that various data protocols can all be processed with the same worker processors on different types of operating systems, which are typically distributed. An embodiment uses a mobile device (such as a mobile phone) as a device and deploys a video sensor application on the mobile device for encoding consecutive video files, time stamping the consecutive video files, and pushing the consecutive video files to a file server to produce a stable stream of video data. Thus avoiding the inefficiencies associated with having video processing in the data flow loop.

Claims

1. A system for activity monitoring using video data, the system comprising:

at least one special purpose processor executing a plurality of video analysis worker processes;

a management subsystem coupled to the at least one special purpose processor that performs specific tasks, wherein the management subsystem comprises;

a feed management subsystem for managing video data from the plurality of video data sources;

a worker management subsystem for managing the video analysis worker processes;

a location management subsystem for managing geographic information related to the plurality of video sources;

a data storage subsystem for managing storage of data, including unprocessed and processed video data; and

a plurality of video data sources of multiple types producing video data of different types, wherein the plurality of video data sources comprises at least one mobile device executing a video sensing application that produces a video stream for processing by the plurality of video analysis worker processes;

wherein the specific tasks include line analysis comprising using video data of people standing in a line to determine how many people wait in line and how long it takes a person to go through the line.

2. A system for activity monitoring using video data, the system comprising:

at least one special purpose processor executing a plurality of video analysis worker processes;

a management subsystem coupled to the at least one special purpose processor that performs specific tasks; and

a plurality of video data sources of multiple types producing video data of different types, wherein the plurality of video data sources comprises at least one mobile device executing a video sensing application that produces a video stream for processing by the plurality of video analysis worker processes, and

a user interface coupled to receive video data from the data storage subsystem and to present the video data in a human-accessible manner;

wherein the specific tasks include line analysis comprising using video data of people standing in a line to determine how many people wait in line and how long it takes a person to go through the line.

3. A system for activity monitoring using video data, the system comprising:

at least one special purpose processor executing a plurality of video analysis worker processes;

a management subsystem coupled to the at least one special purpose processor that performs specific tasks; and

a plurality of video data sources of multiple types producing video data of different types, wherein the plurality of video data sources comprises at least one mobile device executing a video sensing application that produces a video stream for processing by the plurality of video analysis worker processes, a contributor management subsystem for managing entities that control the plurality of video data sources, wherein entities comprise individuals, companies and organizations;

wherein the specific tasks include line analysis comprising using video data of people standing in a line to determine how many people wait in line and how long it takes a person to go through the line.

4. A system for activity monitoring using video data, the system comprising:

at least one special purpose processor executing a plurality of video analysis worker processes, wherein the plurality of video analysis worker processes comprise a process for detection of activity areas;

wherein the process for detection of activity areas comprises;

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate an expected background; and

a management subsystem coupled to the at least one special purpose processor that performs specific tasks; and

a plurality of video data sources of multiple types producing video data of different types, wherein the plurality of video data sources comprises at least one mobile device executing a video sensing application that produces a video stream for processing by the plurality of video analysis worker processes;

wherein the specific tasks include line analysis comprising using video data of a people standing in a line to determine how many people wait in line and how long it takes a person to go through the line.

5. The system of claim 4 wherein:

the process for detection of activity areas comprises;

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate an expected background; and

wherein the plurality of video analysis worker processes comprise a process for detection of activity areas results in activity contours to model a waiting line from the video data.

6. The system of claim 4 wherein:

the process for detection of activity areas comprises;

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate an expected background; and

wherein the plurality of video analysis worker processes comprise a process for detection of activity areas results in activity contours to model a waiting line from the video data, and further results in a curve model of the waiting line using line contours.

7. The system of claim 4 wherein:

the process for detection of activity areas comprises;

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate an expected background; and

wherein the plurality of video analysis worker processes comprise a process for detection of activity areas results in activity contours to model a waiting line from the video data, and further results in a curve model of the waiting line using line contours, and wherein the plurality of video analysis workers generate a measurement of the waiting line in real time using the curve model and the line contours.

8. The system of claim 4 wherein:

the process for detection of activity areas comprises;

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate an expected background; and

wherein the plurality of video analysis worker processes comprise a process for detection of activity areas results in activity contours to model a waiting line from the video data, and further results in a curve model of the waiting line using line contours, and wherein the plurality of video analysis workers generate a measurement of the waiting line in real time using the curve model and the line contours, and wherein the measurement comprises a number of people in the waiting line, and an average wait time.