Methods and systems of performing adaptive morphology operations in video analytics

• US 10,223,590 B2
• Filed: 09/12/2016
• Issued: 03/05/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Wang, Lei, Gao, Jinglun, Bi, Ning
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US15/262,700
• Publication Number:
  US 20180033152A1
• Time in Patent Office: 904 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/30 Erosion or dilatation, e.g. thinning
• CPC Class Codes: G06T 5/75 Unsharp masking
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/155 involving morphological operators
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques and systems are provided for processing video data. For example, techniques and systems are provided for performing content-adaptive morphology operations. A first erosion function can be performed on a foreground mask of a video frame, including setting one or more foreground pixels of the frame to one or more background pixels. A temporary foreground mask can be generated based on the first erosion function being performed on the foreground mask. One or more connected components can be generated for the frame by performing connected component analysis to connect one or more neighboring foreground pixels. A complexity of the frame (or of the foreground mask of the frame) can be determined by comparing a number of the one or more connected components to a threshold number. A second erosion function can be performed on the temporary foreground mask when the number of the one or more connected components is higher than the threshold number. The one or more connected components can be output for blob processing when the number of the one or more connected components is lower than the threshold number.

Claims

1. A method of performing content-adaptive morphology operations, the method comprising:

performing a first erosion function on a foreground mask of a frame, wherein, subsequent to a background subtraction process, the foreground mask includes a binary image of foreground pixels, the first erosion function setting one or more foreground pixels of the foreground mask to one or more background pixels thereby removing the one or more foreground pixels from the foreground mask of the frame;

generating one or more connected components, wherein a connected component connects one or more neighboring foreground pixels of the foreground mask;

determining a complexity of the foreground mask, wherein the complexity of the foreground mask is based on comparing a number of connected components to a threshold number of connected components; and

determining whether to perform one or more additional erosion functions for the frame based on the determined complexity of the foreground mask.

2. The method of claim 1, further comprising:

performing a second erosion function on a temporary foreground mask when the number of connected components is higher than the threshold number of connected components, the temporary foreground mask including the foreground mask after the first erosion function is performed on the foreground mask, and the second erosion function setting one or more foreground pixels of the temporary foreground mask to one or more background pixels.

3. The method of claim 2, further comprising:

generating one or more additional connected components using the temporary foreground mask after the second erosion function is performed;

determining a complexity of the temporary foreground mask after the second erosion function is performed, the complexity of the temporary foreground mask being determined by comparing a number of the one or more additional connected components to the threshold number of connected components;

determining the number of the one or more additional connected components is below the threshold number of connected components; and

outputting the one or more additional connected components for blob processing, wherein a blob includes at least one connected component.

4. The method of claim 2, wherein performing the first erosion function includes performing an erosion operation on the foreground mask, and wherein performing the second erosion function includes re-performing the erosion operation on the temporary foreground mask.

5. The method of claim 2, wherein performing the first erosion function includes performing a first erosion operation on the foreground mask, and wherein performing the second erosion function includes performing a second erosion operation on the temporary foreground mask, the first erosion operation being different than the second erosion operation.

6. The method of claim 1, further comprising:

outputting the one or more connected components for blob processing when the number of connected components is lower than the threshold number of connected components, wherein a blob includes at least one connected component.

7. The method of claim 1, wherein a blob includes at least one connected component, and wherein only connected components of blobs smaller than a size threshold are included in the number of connected components compared to the threshold number of connected components.

8. The method of claim 1, further comprising:

performing at least one dilation function on the foreground mask, the at least one dilation function setting one or more background pixels of the foreground mask to one or more foreground pixels, wherein the at least one dilation function sets a background pixel in the foreground mask to a foreground pixel when at least one neighboring pixel of the background pixel includes a foreground pixel.

9. The method of claim 8, wherein the at least one dilation function is performed after the first erosion function is performed on the foreground mask and before a connected component analysis is performed on the foreground pixels of the foreground mask.

10. The method of claim 8, further comprising:

performing multiple dilation functions on the foreground mask, wherein the multiple dilation functions are performed after the first erosion function is performed on the foreground mask and before a connected component analysis is performed on the foreground pixels of the foreground mask.

11. The method of claim 1, wherein performing the first erosion function includes performing a weak erosion function, the weak erosion function setting a foreground pixel of the foreground mask to a background pixel when two or more neighboring pixels of the foreground pixel include background pixels.

12. The method of claim 1, wherein performing the first erosion function includes setting a foreground pixel of the foreground mask to a background pixel when a single neighboring pixel of includes a background pixel.

13. An apparatus comprising:

a memory configured to store video data; and

a processor configured to;

perform a first erosion function on a foreground mask of a frame, wherein, subsequent to a background subtraction process, the foreground mask includes a binary image of foreground pixels, the first erosion function setting one or more foreground pixels of the foreground mask to one or more background pixels thereby removing the one or more foreground pixels from the foreground mask of the frame;

generate one or more connected components wherein a connected component connects one or more neighboring foreground pixels of the foreground mask;

determine a complexity of the foreground mask, wherein the complexity of the foreground mask is based on comparing a number of connected components to a threshold number of connected components; and

determine whether to perform one or more additional erosion functions for the frame based on the determined complexity of the foreground mask.

14. The apparatus of claim 13, wherein the processor is further configured to:

perform a second erosion function on a temporary foreground mask when the number of connected components is higher than the threshold number of connected components, the temporary foreground mask including the foreground mask after the first erosion function is performed on the foreground mask, and the second erosion function setting one or more foreground pixels of the temporary foreground mask to one or more background pixels.

15. The apparatus of claim 14, wherein the processor is further configured to:

generate one or more additional connected components using the temporary foreground mask after the second erosion function is performed;

determine a complexity of the temporary foreground mask after the second erosion function is performed, the complexity of the temporary foreground mask being determined by comparing a number of the one or more additional connected components to the threshold number of connected components;

determine the number of the one or more additional connected components is below the threshold number of connected components; and

output the one or more additional connected components for blob processing, wherein a blob includes at least one connected component.

16. The apparatus of claim 14, wherein performing the first erosion function includes performing an erosion operation on the foreground mask, and wherein performing the second erosion function includes re-performing the erosion operation on the temporary foreground mask.

17. The apparatus of claim 14, wherein performing the first erosion function includes performing a first erosion operation on the foreground mask, and wherein performing the second erosion function includes performing a second erosion operation on the temporary foreground mask, the first erosion operation being different than the second erosion operation.

18. The apparatus of claim 13, wherein the processor is further configured to:

output the one or more connected components for blob processing when the number of connected components is lower than the threshold number of connected components, wherein a blob includes at least one connected component.

19. The apparatus of claim 13, wherein a blob includes at least one connected component, and wherein only connected components of blobs smaller than a size threshold are included in the number of connected components compared to the threshold number of connected components.

20. The apparatus of claim 13, wherein the processor is further configured to:

perform at least one dilation function on the foreground mask, the at least one dilation function setting one or more background pixels of the foreground mask to one or more foreground pixels, wherein the at least one dilation function sets a background pixel in the foreground mask to a foreground pixel when at least one neighboring pixel of the background pixel includes a foreground pixel.

21. The apparatus of claim 20, wherein the at least one dilation function is performed after the first erosion function is performed on the foreground mask and before a connected component analysis is performed on the foreground pixels of the foreground mask.

22. The apparatus of claim 20, wherein the processor is further configured to:

perform multiple dilation functions on the foreground mask, wherein the multiple dilation functions are performed after the first erosion function is performed on the foreground mask and before a connected component analysis is performed on the foreground pixels of the foreground mask.

23. The apparatus of claim 13, wherein performing the first erosion function includes performing a weak erosion function, the weak erosion function setting a foreground pixel of the foreground mask to a background pixel when two or more neighboring pixels of the foreground pixel include background pixels.

24. The apparatus of claim 13, wherein performing the first erosion function includes setting a foreground pixel of the foreground mask to a background pixel when a single neighboring pixel of the foreground pixel includes a background pixel.

25. A non-transitory computer readable medium having stored thereon instructions that when executed by a processor cause the processor to perform operations including:

performing a first erosion function on a foreground mask of a frame, wherein, subsequent to a background subtraction process, the foreground mask includes a binary image of foreground pixels, the first erosion function setting one or more foreground pixels of the foreground mask to one or more background pixels thereby removing the one or more foreground pixels from the foreground mask of the frame;

generating one or more connected components wherein a connected component connects one or more neighboring foreground pixels of the foreground mask;

determining a complexity of the foreground mask, wherein the complexity of the foreground mask is based on comparing a number of connected components to a threshold number of connected components; and

determining whether to perform one or more additional erosion functions for the frame based on the determined complexity of the foreground mask.

26. The non-transitory computer readable medium of claim 25, further comprising instructions that, when executed by a process, cause the processor to perform operations including:

performing a second erosion function on a temporary foreground mask when the number of connected components is higher than the threshold number of connected components, the temporary foreground mask including the foreground mask after the first erosion function is performed on the foreground mask, and the second erosion function setting one or more foreground pixels of the temporary foreground mask to one or more background pixels.

27. The non-transitory computer readable medium of claim 26, further comprising instructions that, when executed by a process, cause the processor to perform operations including:

generating one or more additional connected components using the temporary foreground mask after the second erosion function is performed;

determining a complexity of the temporary foreground mask after the second erosion function is performed, the complexity of the temporary foreground mask being determined by comparing a number of the one or more additional connected components to the threshold number of connected components;

determining the number of the one or more additional connected components is below the threshold number of connected components; and

outputting the one or more additional connected components for blob processing, wherein a blob includes at least one connected component.

28. The non-transitory computer readable medium of claim 25, further comprising instructions that, when executed by a process, cause the processor to perform operations including:

outputting connected components for blob processing when the number of connected components is lower than the threshold number of connected components, wherein a blob includes at least one connected component.

29. The non-transitory computer readable medium of claim 26, wherein performing the first erosion function includes performing an erosion operation on the foreground mask, and wherein performing the second erosion function includes re-performing the erosion operation on the temporary foreground mask.

30. The non-transitory computer readable medium of claim 26, wherein performing the first erosion function includes performing a first erosion operation on the foreground mask, and wherein performing the second erosion function includes performing a second erosion operation on the temporary foreground mask, the first erosion operation being different than the second erosion operation.