Video segmentation method

• US 10,096,117 B2
• Filed: 12/18/2015
• Issued: 10/09/2018
• Est. Priority Date: 12/24/2014
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Gupta, Amit Kumar, Partis, Ashley John, See, David Kenji
• Primary Examiner(s): Liu, Li
• Application Number:
  US14/975,228
• Publication Number:
  US 20160189388A1
• Time in Patent Office: 1,026 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

A method for classifying a visual element in a video frame as either foreground or background, the video frame being characterized using a scene model comprising a plurality of modes, the method comprising the steps of: determining a blob boundary characteristic for a blob comprising at least the visual element; identifying a mode matched to the visual element; classifying the visual element and the matched mode as foreground dependent upon a match mode boundary characteristic of the matched mode; and updating the scene model dependent upon the blob boundary characteristic and the matched mode boundary characteristic.

Claims

1. A method for classifying a visual element as either foreground or background in a video frame, the method comprising the steps of:

computing a blob boundary characteristic for a blob obtained from the video frame which is captured by a camera module including a processor and a memory storing a computer executable software program for directing the processor to compute the blob boundary characteristic, wherein said blob comprises at least one visual element corresponding to a scene model which comprises a plurality of modes, and said visual element is labelled foreground based on at least a match mode boundary characteristic in the scene model; and

updating the scene model based on at least the computed blob boundary characteristic and the match mode boundary characteristic.

2. A method for classifying a visual element in a video frame as either foreground or background, the video frame being characterised using a scene model comprising a plurality of modes, the method comprising the steps of:

determining a blob boundary characteristic (b_bc) for a blob comprising at least the visual element, the blob being obtained from the video frame which is captured by a camera module including a processor and a memory storing a computer executable software program for directing the processor to determine the blob boundary characteristic;

identifying a mode matched to the visual element;

classifying the visual element as foreground dependent upon a match mode boundary characteristic (m_bc) of the matched mode; and

updating the scene model dependent upon the blob boundary characteristic (b_bc) and the matched mode boundary characteristic (m_bc).

3. The method according to claim 2, wherein determining the blob boundary characteristic comprises determining a gradient across a boundary between the visual element of the blob and a neighbouring visual element.

4. The method according to claim 2, wherein identifying the mode matched to the visual element comprises the steps of:

comparing visual descriptors of modes in the scene model against a visual feature of the visual element;

determining a match score for each comparison; and

identifying the mode matched to the visual element as the mode having the highest match score.

5. The method according to claim 2, wherein classifying the visual element as foreground comprises the steps of:

determining a probability that the visual element is foreground dependent upon a value of the match mode boundary characteristic over time; and

classifying the visual element as foreground dependent upon the probability and a threshold.

6. An apparatus for classifying a visual element in a video frame as either foreground or background, the video frame being characterised using a scene model comprising a plurality of modes, the apparatus comprising:

a camera module for capturing the video frame;

a processor; and

a memory storing a computer executable software program for directing the processor to perform a method comprising the steps of;

determining a blob boundary characteristic for a blob comprising at least the visual element;

identifying a mode matched to the visual element;

classifying the visual element and the matched mode as foreground dependent upon a match mode boundary characteristic of the matched mode; and

updating the scene model dependent upon the blob boundary characteristic and the matched mode boundary characteristic.

7. A non-transitory computer readable memory medium storing a computer executable software program for directing a processor to perform a method for classifying a visual element in a video frame as either foreground or background, the video frame being characterised using a scene model comprising a plurality of modes, the method comprising the steps of:

determining a blob boundary characteristic for a blob comprising at least the visual element;

identifying a mode matched to the visual element;

classifying the visual element and the matched mode as foreground dependent upon a match mode boundary characteristic of the matched mode; and

updating the scene model dependent upon the blob boundary characteristic and the matched mode boundary characteristic.

8. A method for classifying, using a scene model, at least one visual element as either foreground or background in a video frame, the method comprising the steps of:

classifying, using the scene model, a blob including the at least one visual element as either foreground or background, the scene model including a boundary characteristic;

determining a boundary characteristic of the blob classified as foreground; and

updating, on a basis of the determined boundary characteristic of the blob, the boundary characteristic of the scene model corresponding to the at least one visual element included in the blob.

9. The method according to claim 8, wherein determining the boundary characteristic of the blob comprises determining a gradient across a boundary between the visual element included in the blob and a neighbouring visual element not included in the blob.

10. An apparatus for classifying, using a scene model, at least one visual element as either foreground or background in a video frame, the apparatus comprising a memory storing instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform operations comprising:

classifying, using the scene model, a blob including the at least one visual element as either foreground or background, the scene model including a boundary characteristic;

determining a boundary characteristic of the blob classified as foreground; and

updating, on a basis of the determined boundary characteristic of the blob, the boundary characteristic of the scene model corresponding to the at least one visual element included in the blob.

11. A non-transitory computer readable memory medium storing a computer executable software program for directing a processor to perform a method for classifying a visual element in a video frame as either foreground or background, the method comprising the steps of:

classifying, using a scene model, a blob including the at least one visual element as either foreground or background, the scene model including a boundary characteristic;

determining a boundary characteristic of the blob classified as foreground; and

updating, on a basis of the determined boundary characteristic of the blob, the boundary characteristic of the scene model corresponding to the at least one visual element included in the blob.

12. A method for classifying at least one visual element in video frames, the method comprising the steps of:

segmenting an input video frame into a plurality of visual elements based on pixel values;

determining, for each of the plurality of visual elements, a temporal characteristic of a visual element;

determining a boundary characteristic of a blob including the visual element; and

classifying the visual element as either foreground or background based on the temporal characteristic of the visual element and the boundary characteristic of the blob including the visual element.

13. The method according to claim 12, wherein determining the boundary characteristic of the blob comprises determining continuity of pixel values across a boundary of the blob.

14. The method according to claim 13, wherein the temporal characteristic includes a parameter which prescribes a duration time of the visual element.

15. The method according to claim 12, wherein classifying the visual element comprises determining a probability of a case that the visual element belongs to the background, and the probability becomes greater as strength of the continuity of pixel values across the boundary becomes higher.

16. The method according to claim 12, wherein in classifying the visual element, it takes a longer duration time for the visual element to be classified as the background, if strength of the continuity of pixel values across the boundary is high.

17. An apparatus for classifying at least one visual element in video frames, the apparatus comprising a memory storing instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform operations comprising:

segmenting an input video frame into a plurality of visual elements based on pixel values;

determining, for each of the plurality of visual elements, a temporal characteristic of a visual element;

determining a boundary characteristic of a blob including the visual element; and

classifying the visual element as either foreground or background based on the temporal characteristic of the visual element and the boundary characteristic of the blob including the visual element.

18. A non-transitory computer readable memory medium storing a computer executable software program for directing a processor to perform a method for classifying at least one visual element in video frames, the method comprising the steps of:

segmenting an input video frame into a plurality of visual elements based on pixel values;

determining, for each of the plurality of visual elements, a temporal characteristic of a visual element;

determining a boundary characteristic of a blob including the visual element; and

classifying the visual element as either foreground or background based on the temporal characteristic of the visual element and the boundary characteristic of the blob including the visual element.