Computer Vision Pipeline and Methods for Detection of Specified Moving Objects

• US 20160171311A1
• Filed: 12/15/2015
• Published: 06/16/2016
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: Sighthound, Inc.
• Original Assignee: Sighthound, Inc.
• Inventors: Case, Ryan, Masood, Syed Zain, Shu, Guang, Ortiz, Enrique G., Neish, Stephen
• Granted Patent: US 9,710,716 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A computer vision pipeline detects tracks and classifies people or other specified class of objects in a steam of video. The ability to not only detect motion, but to distinguish people or other specified objects, can improve the systems usefulness in applications like security monitoring. A motion detection module provides a motion bitmap and a background subtraction module provides a foreground bitmap, and an object tracking module uses these bitmaps identify and track the specified classes of objects. From these objects and tracks, categorized object data can then be generated.

Claims

1. A method of operating a computer vision system, the method comprising:

receiving a set of vision data at a vision data interface and providing a series of video frames derived from the vision data;

constructing from the series of video frames a motion bitmap at a motion detection module;

constructing from the series of video frames a foreground bitmap at a background subtraction module;

identifying one or more regions within the series of frames each corresponding to an object of a specified category and tracking the one or more regions across multiple frames of the series of video frames by an object tracking module from a comparison of the motion bitmap and foreground bitmap; and

from the tracked regions, determining by an object categorizer module of whether each of the identified regions includes an object of the specified category.

2. The method of claim 1, wherein the regions within the series of video frames are each less than all of a frame.

3. The method of claim 1, further comprising:

generating by the object tracking module of predicted locations within the series of frames for one or more of the identified regions; and

providing the predicted locations to the background subtraction module, wherein the background subtraction module uses the predicted locations in constructing the foreground bitmap.

4. The method of claim 1, wherein providing the series of video frames derived from the vision data including converting a color space format of the vision data.

5. The method of claim 1, wherein the motion bitmap is constructed from a frame by frame difference comparison of the series of video frames.

6. The method of claim 1, wherein the foreground bitmap is constructed using a Gaussian mixture model modeling scheme.

7. The method of claim 1, wherein determining of whether each of the identified regions includes an object of the specified category uses a histogram of oriented gradients technique.

8. The method of claim 1, wherein determining of whether each of the identified regions includes an object of the specified category uses deep learning techniques.

9. The method of claim 1, wherein determining whether each of the identified regions includes an object of the specified category includes:

filtering out of non-random oscillatory motion from the identified regions.

10. The method of claim 1, wherein determining whether each of the identified regions includes an object of the specified category includes:

filtering out of high contrast areas from the identified regions.

11. The method of claim 10, wherein the filtering out of high contrast areas from the identified regions is performed using multiple color spaces.

12. The method of claim 1, wherein determining whether each of the identified regions includes an object of the specified category includes:

determining that a first of the identified regions includes a portion of the object of the specified category, and, in response thereto, relaxing one or more criteria for determining whether the first of the identified regions include the object of the specified category.

13. The method of claim 1, further comprising:

generating by the object categorizer module of category labels for objects of the specified category within the identified regions; and

providing the category labels to object tracking module, wherein the object tracking module uses the category labels in tracking the regions.

14. A computer vision system, comprising:

a vision data interface connectable to receive a set of vision data at and provide a series of video frames derived from the vision data;

a motion detection module connected to the vision data interface to receive the series of video frames and construct therefrom a motion bitmap;

a background subtraction module connected to the vision data interface to receive the series of video frames and construct therefrom a foreground bitmap;

an object tracking module connected to receive the motion bitmap and the foreground bitmap and, from a comparison of the motion bitmap and foreground bitmap, identify one or more regions within the series of frames each corresponding to an object of a specified category and track the one or more regions across multiple frames of the series of video; and

an object categorizer module to determine from the tracked regions whether each of the identified regions includes an object of the specified category.

15. The computer vision system of claim 14, wherein the regions within the series of video frames are each less than all of a frame.

16. The computer vision system of claim 14, wherein the object tracking module generates of predicted locations within the series of frames for one or more of the identified regions and provides the predicted locations to the background subtraction module, wherein the background subtraction module uses the predicted locations in constructing the foreground bitmap.

17. The computer vision system of claim 14, wherein in providing the series of video frames derived from the vision data, the vision data interface converts a color space format of the vision data.

18. The computer vision system of claim 14, wherein the motion bitmap is constructed from a frame by frame difference comparison of the series of video frames.

19. The computer vision system of claim 14, wherein the foreground bitmap is constructed using a Gaussian mixture model modeling scheme.

20. The computer vision system of claim 14, wherein determining of whether each of the identified regions includes an object of the specified category uses a histogram of oriented gradients technique.

21. The computer vision system of claim 14, wherein determining of whether each of the identified regions includes an object of the specified category uses deep learning techniques.

22. The computer vision system of claim 14, wherein determining whether each of the identified regions includes an object of the specified category includes filtering out of non-random oscillatory motion from the identified regions.

23. The computer vision system of claim 14, wherein determining whether each of the identified regions includes an object of the specified category includes filtering out of high contrast areas from the identified regions.

24. The computer vision system of claim 23, wherein the filtering out of high contrast areas from the identified regions is performed using multiple color spaces.

25. The computer vision system of claim 14, wherein determining whether each of the identified regions includes an object of the specified category includes determining that a first of the identified regions includes a portion of the object of the specified category, and, in response thereto, relaxing one or more criteria for determining whether the first of the identified regions include the object of the specified category.

26. The computer vision system of claim 14, wherein the object categorizer module generates category labels for objects of the specified category within the identified regions and provides the category labels to object tracking module, wherein the object tracking module uses the category labels in tracking the regions.