Object-level grouping and identification for tracking objects in a video

• US 9,898,677 B1
• Filed: 10/10/2016
• Issued: 02/20/2018
• Est. Priority Date: 10/13/2015
• Status: Active Grant

• Current Assignee: Cubic Corporation
• Original Assignee: MotionDSP, Inc. (Cubic Corporation)
• Inventors: Markovic, Milan, Grujic, Nemanja, Ilic, Vuk, Mulalic, Edin, Andelkovic, Sa{hacek over (s)}a, Andjelkovic, Neboj{hacek over (s)}a
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US15/289,888
• Time in Patent Office: 498 Days
• Field of Search: 382103, 382224
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

In one embodiment, a method of determine and track moving objects in a video, including detecting and extracting regions from accepted frames of a video, matching parts including using the extracted parts of a current frame and matching each part from a previous frame to a region in the current frame, tracking the matched parts to form part tracks, and determining a set of path features for each tracked part path. The determined path features are used to classify each path as that of mover or a static. The method includes clustering the paths of movers, including grouping parts of movers that likely belong to a single object, in order to generate one or more single moving objects and track moving objects. Also a system to carry out the method and a non-transitory computer-readable medium that when executed in a processing system causes carrying out the method.

Claims

1. A method of operating a hardware system to process a video that includes an ordered sequence of frames, the method to determine and track moving objects in the video, the method comprising:

accepting frames of a video from a source of video data;

detecting and extracting regions from the accepted frames, each region possibly being a part of an object;

matching parts including using the extracted parts of a current frame and matching each part from a previous frame to a region in the current frame, a previous frame being a frame previous to the current frame in the sequence;

tracking the matched parts to form part tracks, the tracking being in relation to one or more most recent reference frames, the tracking comprising changing reference frames from time to time;

determining a set of path features for each tracked part path, and using the determined path features to classify each path as that of a moving part (a mover) or a stationary path (a static); and

clustering the paths of movers, including grouping parts of movers that likely belong to a single object, in order to generate one or more single moving objects, and in order to track the single moving objects.

2. The method of claim 1, wherein the source of video data comprises at least one of a video camera, a computer-readable file, and a packetized video stream.

3. The method of claim 1, wherein the extracted regions are extremal regions.

4. The method of claim 3, wherein the extremal regions are each a maximal stable extremal region (MSER).

5. The method of claim 1, wherein the changing of reference frames from time to time is periodic with a change-reference-frame period.

6. The method of claim 1, wherein the tracking comprises from time to time introducing new parts to track that were not previously matched.

7. The method of claim 6, wherein the introducing new parts from time to time is periodic with a new-part-period.

8. The method of claim 1, wherein the clustering comprises for the current frame, determining a set of connectedness features that capture the relationship between pairs of movers in the current frame.

9. The method of claim 8, wherein the connectedness features include at least two features of the set consisting of features related to a velocity difference of pairs of movers, features related to proximity of the mover parts, features related to overlap between the mover parts, and features related to background similarity of respective backgrounds of the mover parts.

10. The method of claim 9, wherein the connectedness features are normalized, such that the method robust to scale.

11. The method of claim 8, wherein the clustering includes estimating two sets of probability functions for the connectedness features, a first set of probability functions for features of pairs belonging to the same part and a second set of probability functions for features of pairs not belonging.

12. The method of claim 11, wherein estimating the probability function sets are comprises carrying out supervised learning using a training set of videos containing static and moving parts that have pre-labeled into moving and non-moving objects.

13. The method of claim 11, wherein the clustering comprises using the estimated probability functions to determine a measure of probability of any two pairs of parts belonging to the same object.

14. The method of claim 1, wherein the clustering comprises matching of moving objects between frames.

15. The method of claim 14, wherein the matching of moving objects between frames comprises for a candidate object in the current frame determining one or more correspondences between with one or more objects in the previous frame, if there is at least one object in the previous frame having a correspondence with the candidate object.

16. The method of claim 15, wherein the matching of moving objects comprises determining a measure of correspondence from candidate groupings in the current frame to past object groupings based on the numbers of parts in a candidate object in the current frame that are in objects that contain the respective parts in one or more previous frames.

17. The method of claim 1, wherein the clustering comprises carrying out bipartite matching to find correspondences for propagation of object groupings from frame to frame.

18. The method of claim 1, wherein the clustering comprises short interval grouping (SIG) filtering that may eliminate one or more short interval erroneous groupings.

19. A system for processing video that includes an ordered sequence of frames to determine and track moving objects in the video, the system comprising:

an input port operative to accept video frames from a source of video frames;

a region-extractor operative to detect and extracting regions from the accepted frames, each region possibly being a part of an object;

a part matcher operative to match parts including using the extracted parts of a current frame and matching each part from a previous frame to a region in the current frame, a previous frame being a frame previous to the current frame in the sequence;

a part tracker operative to track the matched parts to form part tracks, the tracking being in relation to one or more most recent reference frames, the tracking comprising changing reference frames from time to time;

a path feature classifier operative to determine a set of path features for each tracked part path, and to use the determined path features to classify each path as that of a moving part (a mover) or a stationary path (a static); and

a clusterer operative to cluster the paths of movers, including grouping parts of movers that likely belong to a single object, in order to generate one or more single moving objects, and in order to track the single moving objects.

20. The system of claim 19, wherein the source of video data comprises at least one of a video camera, a computer-readable file, and a packetized video stream.

21. The system of claim 19, wherein the extracted regions are extremal regions.

22. The system of claim 21, wherein the extremal regions are each a maximal stable extremal region (MSER).

23. The system of claim 19, wherein the changing of reference frames from time to time is periodic with a change-reference-frame period.

24. The system of claim 19, wherein the tracking comprises from time to time introducing new parts to track that were not previously matched.

25. The system of claim 24, wherein the introducing new parts from time to time is periodic with a new-part-period.

26. The system of claim 19, wherein the clustering by the clusterer comprises for the current frame, determining a set of connectedness features that capture the relationship between pairs of movers in the current frame.

27. The system of claim 26, wherein the connectedness features include at least two features of the set consisting of features related to a velocity difference of pairs of movers, features related to proximity of the mover parts, features related to overlap between the mover parts, and features related to background similarity of respective backgrounds of the mover parts.

28. The system of claim 27, wherein the connectedness features are normalized, such that the method robust to scale.

29. The system of claim 25, wherein the clustering includes estimating two sets of probability functions for the connectedness features, a first set of probability functions for features of pairs belonging to the same part and a second set of probability functions for features of pairs not belonging.

30. The system of claim 29, wherein estimating the probability function sets are comprises carrying out supervised learning using a training set of videos containing static and moving parts that have pre-labeled into moving and non-moving objects.

31. The system of claim 29, wherein the clustering comprises using the estimated probability functions to determine a measure of probability of any two pairs of parts belonging to the same object.

32. The system of claim 19, wherein the clustering comprises matching of moving objects between frames.

33. The system of claim 32, wherein the matching of moving objects between frames comprises for a candidate object in the current frame determining one or more correspondences between with one or more objects in the previous frame, if there is at least one object in the previous frame having a correspondence with the candidate object.

34. The system of claim 33, wherein the matching of moving objects comprises determining a measure of correspondence from candidate groupings in the current frame to past object groupings based on the numbers of parts in a candidate object in the current frame that are in objects that contain the respective parts in one or more previous frames.

35. The system of claim 19, wherein the clustering comprises carrying out bipartite matching to find correspondences for propagation of object groupings from frame to frame.

36. The system of claim 19, wherein the clustering comprises short interval grouping (SIG) filtering that may eliminate one or more short interval erroneous groupings.

37. The system of claim 19, comprising one or more processors and a storage subsystem, the storage subsystem including instructions that when executed by one or more of the processors, carry out the operations of the region-extractor, the part matcher, the part tracker, the part feature classifier, and the clusterer.

38. A non-transitory machine-readable medium comprising instructions that when executed by one or more processors of a processing system coupled to a source of video data carry out a method of processing a video that includes an ordered sequence of frames, the method to determine and track moving objects in the video, the method comprising:

accepting frames of the video from the source of video data;

detecting and extracting regions from the accepted frames, each region possibly being a part of an object;

matching parts including using the extracted parts of a current frame and matching each part from a previous frame to a region in the current frame, a previous frame being a frame previous to the current frame in the sequence;

tracking the matched parts to form part tracks, the tracking being in relation to one or more most recent reference frames, the tracking comprising changing reference frames from time to time;

determining a set of path features for each tracked part path, and using the determined path features to classify each path as that of a moving part (a mover) or a stationary path (a static); and

clustering the paths of movers, including grouping parts of movers that likely belong to a single object, in order to generate one or more single moving objects, and in order to track the single moving objects.