Systems and methods for processing video streams

• US 10,380,431 B2
• Filed: 10/07/2016
• Issued: 08/13/2019
• Est. Priority Date: 06/01/2015
• Status: Active Grant

• Current Assignee: Placemeter, LLC
• Original Assignee: Placemeter, LLC
• Inventors: Winter, Alexandre, Jardonnet, Ugo, Thi, Tuan Hue, Haldimann, Niklaus
• Primary Examiner(s): Sosanya, Obafemi O
• Application Number:
  US15/288,085
• Publication Number:
  US 20170070707A1
• Time in Patent Office: 1,040 Days
• CPC Class Codes: G06F 18/21355 nonlinear criteria, e.g. embedding a manifold in a Euclidean space
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20044 Skeletonization; Medial axis transform
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/182 Network patterns, e.g. roads or rivers
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

Embodiments of a method and system described herein enable capture of video data streams from multiple, different video data source devices and the processing of the video data streams. The video data streams are merged such that various data protocols can all be processed with the same worker processors on different types of operating systems, which are typically distributed.

Claims

1. A computer-implemented method for processing video streams, the method comprising:

a special purpose processor collecting video data from a plurality of dissimilar sources, that manages video feeds from the plurality of dissimilar sources, processes the video feeds into data, stores the data, and computes analytics and predictions;

The system includes a backend subsystem consisting of specially programmed processors executing software that manages video feeds, processes the video feeds into data, stores the data, and computes analytics and predictions

the special purpose processor, in an iterative process, defining known foreground objects and newly emerged objects;

the special purpose processor determining moving objects from the video data, wherein determining moving objects comprises,

movement detection comprising at least one non-linear time-domain high-pass filter, and determining trajectories of objects;

foreground detection comprising processing video data from a current frame and an expected background, including determining an absolute difference between the current frame and an expected background, and adding R, G and B planes with saturation to generate a merged absolute difference with background;

object tracking comprising determining histories of the positions of objects;

classifying moving objects comprising receiving groups of foreground pixels and outputting one or more objects per group with an associated class using class supervised learning; and

high level analysis comprising determining one or more of, the speed of an object, whether an object is entering or leaving a building, and whether an object is crossing a virtual turnstile; and

the special purpose processor processing the video data as video streams from the plurality of dissimilar sources, providing video data of different types, to allow the video data from the plurality of dissimilar sources to be analyzed without dependence on a source of any of the video data.

2. The method of claim 1, further comprising the special purpose processor:

representing all starting points and ending points of trajectories on a map of the video streams; and

performing a local window analysis of a geographic distribution of the starting points and ending points.

3. The method of claim 1, wherein the video data comprises a plurality of images, and wherein the image data is subsampled by an integer factor to facilitate processing speed.

4. The method of claim 1, wherein the video data comprises a plurality of images, and wherein determining moving objects comprises determining background statistics, wherein background statistics can be updated at variable intervals to affect different processing speeds.

5. The method of claim 1, wherein determining moving objects from the video data comprises estimating the background image of a fixed video stream, comprising modeling points in the video data using a Gaussian distribution of values on each channel of a color image.

6. The method of claim 1, wherein tracking the moving objects comprises receiving an instance of an object at a point in time and outputting a linked appearance of the same object at different times, with trajectory and shape over time.

7. The method of claim 1, wherein analyzing the video data comprises receiving objects with trajectories and outputting information regarding the objects, comprising speed, of objects, objects entering or leaving a location, and inference of building layouts based on movement of objects.

8. A non-transient computer readable medium having stored thereon instructions, that when executed by a special purpose processor cause a method for processing video data to be performed, the method comprising:

a special purpose processor collecting video data from a plurality of dissimilar sources, that manages video feeds from the plurality of dissimilar sources, processes the video feeds into data, stores the data, and computes analytics and predictions;

the special purpose processor, in an iterative process, defining known foreground objects and newly emerged objects;

the special purpose processor determining moving objects from the video data;

the special purpose processor classifying the moving objects from the video data comprising receiving groups of foreground pixels and outputting one or more objects per group with an associated class using class supervised learning;

the special purpose processor tracking the moving objects from the video data wherein tracking comprises receiving as input an instance of one object at one point in time and calculating its trajectory and shape over time, and for each new video frame attempting to match foreground objects with existing objects tracked in prior iterations of the iterative process, and determining where the moving objects start and end their trajectories over a period of time;

the special purpose processor determining likely detection of a building entrance based on where the moving objects start and end their trajectories over the period of time; and

the special purpose processor processing the video data as video streams from the plurality of dissimilar sources, providing video data of different types, to allow the video data from the plurality of dissimilar sources to be analyzed without dependence on a source of any of the video data.

9. The non-transient computer readable medium of claim 8, wherein the video data comprises a plurality of images, and wherein the image data is subsampled by an integer factor to facilitate processing speed.

10. The non-transient computer readable medium of claim 8, wherein the video data comprises a plurality of images, and wherein determining moving objects comprises determining background statistics, wherein background statistics can be updated at variable intervals to affect different processing speeds.

11. The non-transient computer readable medium of claim 8, wherein determining moving objects from the video data comprises estimating the background image of a fixed video stream, comprising modeling points in the video data using a Gaussian distribution of values on each channel of a color image.

12. The non-transient computer readable medium of claim 8, wherein classifying moving objects comprises receiving groups of foreground pixels and outputting one or more objects per group with an associated class using class supervised learning.

13. The non-transient computer readable medium of claim 8, wherein tracking the moving objects comprises receiving an instance of an object at a point in time and outputting a inked appearance of the same object at different times, with trajectory and shape over time.

14. The non-transient computer readable medium of claim 8, wherein analyzing the video data comprises receiving objects with trajectories and outputting information regarding the objects, comprising speed, of objects, objects entering or leaving a location, and inference of building layouts based on movement of objects.

15. A system for processing multiple video data streams, comprising:

a plurality of input video data sources;

a backend subsystem configured to receive video data from the plurality of dissimilar input video data sources, managing video feeds from the plurality of dissimilar sources, processing the video feeds into data, storing the data, and computing analytics and predictions, the backend subsystem comprising,

a plurality of video analysis workers comprising special purpose processors tasked with executing video analysis worker processes, comprising,

in an iterative process, defining known foreground objects and newly emerged objects;

determining moving objects from the video data;

classifying the moving objects from the video data;

tracking the moving objects from the video data, wherein tracking comprises receiving as input an instance of one object at one point in time and calculating its trajectory and shape over time, and for each new video frame attempting to match foreground objects with existing objects tracked in prior iterations of the iterative process;

classifying moving objects comprising receiving groups of foreground pixels and outputting one or more objects per group with an associated class using class supervised learning;

wherein tracking the moving objects comprises tracking moving objects from a plurality of dissimilar sources, which supply video data of different types, to allow the video data from the plurality of dissimilar sources to be analyzed without dependence on a source of any of the video data; and

a data analytics module configured to receive processed video data from the plurality of video analysis workers and output human readable information, wherein the human readable information comprises likely locations of building entrances.

16. The system of claim 15, wherein the data analytics module is further configured to access external data sources, including weather data sources, and event information sources.

17. The system of claim 15, further comprising at least one user interface configured to provide the human readable information to a user any computing device with processing capability, communication capability, and display capability, including personal computers and mobile devices.

18. The system of claim 15, wherein the special purpose processors determine moving objects from the video data, and wherein determining comprises:

the special purpose processors classifying the moving objects from the video data; and

the special purpose processors tracking the moving objects from the video data.