Methods and systems for shape adaptation for merged objects in video analytics

• US 10,553,091 B2
• Filed: 07/20/2017
• Issued: 02/04/2020
• Est. Priority Date: 03/31/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Bi, Ning, Zhou, Yang
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Rivera-Martinez, Guillermo M
• Application Number:
  US15/655,065
• Publication Number:
  US 20180286199A1
• Time in Patent Office: 929 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19604 : involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position

Abstract

Methods, apparatuses, and computer-readable media are provided for splitting one or more merged blobs for one or more video frames. For example, a merged blob detected for a current video frame is identified. The merged blob includes pixels of at least a portion of at least two foreground objects in the current video frame. The merged blob is associated with a first blob tracker and a second blob tracker. A shape of the first blob tracker can be adjusted. For instance, adjusting the shape of the first blob tracker can include shifting at least one boundary of a bounding region of the first blob tracker based on the shape of the merged blob. The merged blob can be split into a first blob and a second blob, with the first blob being associated with the adjusted bounding region of the first blob tracker and the second blob being associated with a bounding region of the a second blob tracker. The first blob and the second blob can then be output for object tracking for the current video frame.

Claims

1. A method of splitting one or more merged blobs for one or more video frames, the method comprising:

identifying a merged blob detected for a current video frame, wherein the merged blob includes pixels of at least a portion of at least two foreground objects in the current video frame;

associating the merged blob with a first blob tracker and a second blob tracker, wherein the first blob tracker includes a first bounding region and the second blob tracker includes a second bounding region;

adjusting a shape of the first blob tracker, wherein adjusting the shape of the first blob tracker includes shifting at least one boundary of the first bounding region relative to a boundary of the merged blob to generate an adjusted first bounding region;

splitting the merged blob into a first blob and a second blob, the first blob being contained within the adjusted first bounding region and the second blob being contained within the second bounding region; and

outputting the first blob and the second blob obtained from splitting the merged blob, for object tracking for the current video frame.

2. The method of claim 1, wherein shifting the at least one boundary of the first bounding region includes:

shifting the at least one boundary of the first bounding region towards a center of the first bounding region, wherein the at least one boundary is shifted until the at least one boundary intersects with a foreground pixel of the merged blob.

3. The method of claim 1, wherein shifting the at least one boundary of the first bounding region includes:

performing a first shape adjustment by shifting at least one boundary pair of the first blob tracker towards a center of the first bounding region, the at least one boundary pair including the at least one boundary, wherein the at least one boundary pair is shifted until one or more boundaries of the at least one boundary pair intersect with a foreground pixel of the merged blob.

4. The method of claim 3, further comprising:

maintaining a list of historical merge groups, the historical merge groups including associations between merged blobs and blob trackers associated with the merged blobs across one or more previous video frames;

searching the maintained list of historical merge groups to identify a historical merge group that corresponds to a preliminary association of the merged blob, the preliminary association including at least the first bounding region associated with the merged blob and the second bounding region associated with the merged blob; and

performing the first shape adjustment when a historical merge group corresponding to the preliminary association of the merged blob is not identified.

5. The method of claim 3, wherein the at least one boundary pair of the first blob tracker includes at least one or more of a left boundary and a right boundary of the first bounding region or a top boundary and a bottom boundary of the first bounding region.

6. The method of claim 3, further comprising:

determining, before the shape of the first blob tracker is adjusted, a first boundary and a second boundary of the at least one boundary pair do not intersect with any foreground pixel of the merged blob; and

wherein shifting the at least one boundary pair includes shifting the first boundary and the second boundary towards the center of the first bounding region when the first boundary and the second boundary do not intersect with any foreground pixel of the merged blob before the shape of the first blob tracker is adjusted.

7. The method of claim 3, further comprising:

determining, before the shape of the first blob tracker is adjusted, a first boundary of the at least one boundary pair intersects with a foreground pixel of the merged blob;

determining, before the shape of the first blob tracker is adjusted, a second boundary of the at least one boundary pair does not intersect with any foreground pixel of the merged blob; and

wherein shifting the at least one boundary pair includes shifting the second boundary towards the center of the first bounding region when the first boundary intersects with the foreground pixel and the second boundary does not intersect with any foreground pixel of the merged blob before the shape of the first blob tracker is adjusted.

8. The method of claim 1, wherein adjusting the shape associated with the merged blob further includes:

performing a second shape adjustment by individually shifting each boundary of the first bounding region towards the center of the first bounding region, wherein each boundary is shifted until each boundary intersects with a foreground pixel of the merged blob.

9. The method of claim 8, further comprising:

determining the shape of the first blob tracker was adjusted in a previous video frame, the previous video frame being obtained earlier in time than the current video frame; and

performing the second shape adjustment when the shape of the first blob tracker was adjusted in the previous video frame.

10. The method of claim 8, further comprising:

determining a size of the first bounding region is less than a size threshold; and

performing the second shape adjustment when the size of the first bounding region is less than the size threshold.

11. The method of claim 8, further comprising:

determining a size ratio, the size ratio including a size of the first bounding region over a size of a bounding region of the merged blob;

determining the size ratio is less than a size ratio threshold; and

performing the second shape adjustment when the size ratio is less than the size ratio threshold.

12. The method of claim 8, further comprising:

determining a size of the first bounding region is less than a size threshold;

determining a size ratio, the size ratio including a size of the first bounding region over a size of a bounding region of the merged blob;

determining the size ratio is less than a size ratio threshold; and

performing the second shape adjustment when the size of the first bounding region is less than the size threshold and when the size ratio is less than the size ratio threshold.

13. The method of claim 1, further comprising:

obtaining a background-foreground mask for the current frame, the background-foreground mask including a first value assigned to background pixels of the current frame and a second value assigned to foreground pixels of the current frame; and

modifying the background-foreground mask by assigning a common value to each foreground pixel of the merged blob, wherein the common value is different than the first value and the second value.

14. The method of claim 13, wherein shifting the at least one boundary of the first bounding region includes:

shifting the at least one boundary of the first bounding region towards a center of the first bounding region, wherein the at least one boundary is shifted until the at least one boundary intersects with a foreground pixel having the common value.

15. The method of claim 1, wherein associating the merged blob with the first blob tracker and the second blob tracker includes:

determining an intersection region between the first bounding region and a bounding region of the merged blob;

determining a size ratio, the size ratio including a size of the intersection region over a size of the bounding region of the merged blob;

determining the size ratio is greater than a minimum size ratio; and

determining the first blob tracker is associated with the merged blob when the size ratio is greater than the minimum size ratio.

16. The method of claim 15, further comprising:

re-associating the merged blob with the first blob tracker and the second blob tracker after the shape of the first blob tracker is adjusted, wherein the re-association is performed using a second size ratio that is less than the size ratio.

17. An apparatus for splitting one or more merged blobs for one or more video frames, comprising:

a memory configured to store video data associated with one or more video frames; and

a processor configured to;

identify a merged blob detected for a current video frame, wherein the merged blob includes pixels of at least a portion of at least two foreground objects in the current video frame;

associate the merged blob with a first blob tracker and a second blob tracker, wherein the first blob tracker includes a first bounding region and the second blob tracker includes a second bounding region;

adjust a shape of the first blob tracker, wherein adjusting the shape of the first blob tracker includes shifting at least one boundary of the first bounding region relative to a boundary of the merged blob to generate an adjusted first bounding region;

split the merged blob into a first blob and a second blob, the first blob being contained within the adjusted first bounding region and the second blob being contained within the second bounding region; and

output the first blob and the second blob obtained from splitting the merged blob, for object tracking for the current video frame.

18. The apparatus of claim 17, wherein adjusting the shape of the first blob tracker includes:

shifting the at least one boundary of the first bounding region towards a center of the first bounding region, wherein the at least one boundary is shifted until the at least one boundary intersects with a foreground pixel of the merged blob.

19. The apparatus of claim 17, wherein adjusting the shape of the first blob tracker includes:

performing a first shape adjustment by shifting at least one boundary pair of the first blob tracker towards a center of the first bounding region, the at least one boundary pair including the at least one boundary, wherein the at least one boundary pair is shifted until one or more boundaries of the at least one boundary pair intersect with a foreground pixel of the merged blob.

20. The apparatus of claim 19, wherein the processor is further configured to:

maintain a list of historical merge groups, the historical merge groups including associations between merged blobs and blob trackers associated with the merged blobs across one or more previous video frames;

search the maintained list of historical merge groups to identify a historical merge group that corresponds to a preliminary association of the merged blob, the preliminary association including at least the first bounding region associated with the merged blob and the second bounding region associated with the merged blob; and

perform the first shape adjustment when a historical merge group corresponding to the preliminary association of the merged blob is not identified.

21. The apparatus of claim 19, wherein the processor is further configured to:

determine, before the shape of the first blob tracker is adjusted, a first boundary and a second boundary of the at least one boundary pair do not intersect with any foreground pixel of the merged blob; and

wherein shifting the at least one boundary pair includes shifting the first boundary and the second boundary towards the center of the first bounding region when the first boundary and the second boundary do not intersect with any foreground pixel of the merged blob before the shape of the first blob tracker is adjusted.

22. The apparatus of claim 19, wherein the processor is further configured to:

determine, before the shape of the first blob tracker is adjusted, a first boundary of the at least one boundary pair intersects with a foreground pixel of the merged blob;

determine, before the shape of the first blob tracker is adjusted, a second boundary of the at least one boundary pair does not intersect with any foreground pixel of the merged blob; and

wherein shifting the at least one boundary pair includes shifting the second boundary towards the center of the first bounding region when the first boundary intersects with the foreground pixel and the second boundary does not intersect with any foreground pixel of the merged blob before the shape of the first blob tracker is adjusted.

23. The apparatus of claim 17, wherein adjusting the shape associated with the merged blob further includes:

performing a second shape adjustment by individually shifting each boundary of the first bounding region towards the center of the first bounding region, wherein each boundary is shifted until each boundary intersects with a foreground pixel of the merged blob.

24. The apparatus of claim 23, wherein the processor is further configured to:

determine the shape of the first blob tracker was adjusted in a previous video frame, the previous video frame being obtained earlier in time than the current video frame; and

perform the second shape adjustment when the shape of the first blob tracker was adjusted in a previous video frame.

25. The apparatus of claim 23, wherein the processor is further configured to:

determine a size of the first bounding region is less than a size threshold; and

perform the second shape adjustment when the size of the first bounding region is less than the size threshold.

26. The apparatus of claim 23, wherein the processor is further configured to:

determine a size ratio, the size ratio including a size of the first bounding region over a size of a bounding region of the merged blob;

determine the size ratio is less than a size ratio threshold; and

perform the second shape adjustment when the size ratio is less than the size ratio threshold.

27. The apparatus of claim 23, wherein the processor is further configured to:

determine a size of the first bounding region is less than a size threshold;

determine a size ratio, the size ratio including a size of the first bounding region over a size of a bounding region of the merged blob;

determine the size ratio is less than a size ratio threshold; and

perform the second shape adjustment when the size of the first bounding region is less than the size threshold and when the size ratio is less than the size ratio threshold.

28. The apparatus of claim 17, wherein the processor is further configured to:

obtain a background-foreground mask for the current frame, the background-foreground mask including a first value assigned to background pixels of the current frame and a second value assigned to foreground pixels of the current frame; and

modify the background-foreground mask by assigning a common value to each foreground pixel of the merged blob, wherein the common value is different than the first value and the second value.

29. The apparatus of claim 28, wherein adjusting the shape of the first blob tracker includes:

shifting the at least one boundary of the first bounding region towards a center of the first bounding region, wherein the at least one boundary is shifted until the at least one boundary intersects with a foreground pixel having the common value.

30. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to:

identify a merged blob detected for a current video frame, wherein the merged blob includes pixels of at least a portion of at least two foreground objects in the current video frame;

associate the merged blob with a first blob tracker and a second blob tracker, wherein the first blob tracker includes a first bounding region and the second blob tracker includes a second bounding region;

adjust a shape of the first blob tracker, wherein adjusting the shape of the first blob tracker includes shifting at least one boundary of the first bounding region relative to a boundary of the merged blob to generate an adjusted first bounding region;

split the merged blob into a first blob and a second blob, the first blob being contained within the adjusted first bounding region and the second blob being contained within the second bounding region; and

output the first blob and the second blob obtained from splitting the merged blob, for object tracking for the current video frame.