SYSTEM AND METHOD FOR ACTIVITY MONITORING USING VIDEO DATA

• US 20150350608A1
• Filed: 06/01/2015
• Published: 12/03/2015
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Placemeter, Inc. (Arlo Technologies, Inc.)
• Original Assignee: Placemeter, Inc. (Arlo Technologies, Inc.)
• Inventors: WINTER, Alexandre, HALDIMANN, Niklaus, THI, Tuan Hue
• Granted Patent: US 10,432,896 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Embodiments of a method and system described herein enable capture of video data streams from multiple, different devices and the processing of the video data streams. The video data streams are merged such that various data protocols can all be processed with the same worker processors on different types of operating systems, which are typically distributed. An embodiment uses a mobile device (such as a mobile phone) as a device and deploys a video sensor application on the mobile device for encoding consecutive video files, time stamping the consecutive video files, and pushing the consecutive video files to a file server to produce a stable stream of video data. Thus avoiding the inefficiencies associated with having video processing in the data flow loop.

Claims

1. A system for activity monitoring using video data, the system comprising:

at least one special purpose processor executing a plurality of video analysis worker processes;

a management subsystem coupled to the at least one special purpose processor; and

as plurality of video data sources of multiple types producing video data of different types, wherein the plurality of video data sources comprises at least one mobile device executing a video sensing application that produces a video stream for processing by the video analysis worker processes.

2. The system of claim 1, wherein the management subsystem comprises:

a feed management subsystem for managing video data from the plurality of video data sources;

is worker management subsystem for managing the video analysis worker processes;

a location management subsystem for managing geographic information related to the plurality of video sources; and

a data storage subsystem for managing storage of data including unprocessed and processed video data.

3. The system of claim 1, further comprising a user interface coupled to receive video data from the data storage subsystem and to present the video data in a human-accessible manner.

4. The system of claim 1, further comprising a contributor management subsystem for managing entities that control the plurality of video data sources, wherein entities comprise individuals, companies and organizations.

5. The system of claim 1, wherein the video analysis worker processes comprise a process for detection of activity areas.

6. The system of claim 5, wherein the process for detection of activity areas comprises:

detecting movement in a current video frame;

detecting foreground in a current video frame; and

generating and updating a background model to generate and expected background.

7. The system of claim 6, wherein the video analysis worker processes comprise a process for detection of activity areas results in activity contours to model a waiting line from the video data.

8. The system of claim 7, wherein the activity contours are operated on to generate a curve model of the waiting line and line contours.

9. The system of claim 8, wherein the curve model and the line contours are operated on to measure attributes of the waiting line in real time.

10. The system of claim 9, wherein the attributes comprise a number of people in the waiting line, and an average wait time.

11. A computer implemented video data processing method, comprising:

as plurality of special purpose video analysis processors receiving a plurality of video data streams from multiple sources of different types in different locations that generate different types of video data streams;

the plurality of special purpose video analysis processors executing a plurality of video analysis worker processes that operate on the video data streams to generate processed video data that includes real time information related to scenes; and

executing as video sensor application on a mobile device which is one of the multiple sources, wherein the video sensor application performs various functions using the mobile device, the functions comprising,

encoding consecutive video files;

time stamping the consecutive video files; and

pushing the consecutive video files to a file server to produce a stable stream of video data.

12. The method of claim 11, further comprising the video analysis worker processes unifying geo location and time reference information from the plurality of video data streams such that processed video data having similar geolocation and time reference information generated from the multiple sources can be compared.

13. The method of claim 11, further comprising the video analysis worker processes:

gathering and storing processed data from the plurality of video streams;

architecturally removing any dependency of the special purpose video analysis processors on the different types of video data streams;

merging the video data from the multiple sources; and

packaging and serving processed data as an output.

14. The method of claim 11, wherein the video analysis worker processes are accessible via a plurality of application programming interfaces such that the video analysis worker processes instantiated on and executable on a plurality of operating systems, wherein the operating systems comprise Linux, Mac OS, and Windows.

15. The method of claim 14, wherein the video analysis worker processes report level of central processing unit (CPU) activity, availability, and data storage capacity.

16. A computer-readable medium having instructions stored thereon, that when executed by a processor cause a video processing method to be performed, the method comprising:

executing a plurality of video analysis worker processes that operate on a plurality of video data streams to generate processed video data that includes real time information related to scenes, wherein the multiple video streams originate from a plurality of devices of different types, wherein the video analysis worker processes,

detect moving objects based on estimating a background image of a fixed video stream;

classify the moving objects as one of “

known object” and

“

noise; and

track a trajectory of one object over time.

17. The computer-readable medium of claim 16, wherein the method further comprises the video analysis worker processes operate on only moving object video data to detect human readable data comprising vehicle speed, pedestrian speed, humans entering a location, and humans exiting a location.

18. The computer-readable medium of claim 16, wherein the method further comprises the video analysis worker processes:

gathering and storing processed data from the plurality of video streams;

architecturally removing any dependency of the processors on the different types of video data streams;

merging the video data from the plurality of devices; and

packaging and serving processed data as an output.

19. The computer-readable medium of claim 16, wherein the method further comprises executing a video sensor application on a mobile device which is one of the plurality of devices, wherein the video sensor application performs various functions using the mobile device, the functions comprising,

encoding consecutive video files;

time stamping the consecutive video files; and

pushing the consecutive video files to a file server to produce a stable stream of video data.

20. The computer-readable medium of claim 16, wherein the method further comprises the video analysis worker processes make the processed data available to one or more user interfaces that display graphical information and text information regarding a scene in real time.