Inter-mode region-of-interest video object segmentation

• US 20070183662A1
• Filed: 02/07/2006
• Published: 08/09/2007
• Est. Priority Date: 02/07/2006
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wang, Haohong, El-Maleh, Khaled
• Granted Patent: US 8,265,392 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

The disclosure is directed to techniques for automatic segmentation of a region-of-interest (ROI) video object from a video sequence. ROI object segmentation enables selected ROI or “foreground” objects of a video sequence that may be of interest to a viewer to be extracted from non-ROI or “background” areas of the video sequence. Examples of a ROI object are a human face or a head and shoulder area of a human body. The disclosed techniques include a hybrid technique that combines ROI feature detection, region segmentation, and background subtraction. In this way, the disclosed techniques may provide accurate foreground object generation and low-complexity extraction of the foreground object from the video sequence. A ROI object segmentation system may implement the techniques described herein. In addition, ROI object segmentation may be useful in a wide range of multimedia applications that utilize video sequences, such as video telephony applications and video surveillance applications.

Claims

1. A method comprising:

receiving a video frame of a video sequence;

identifying a moving foreground region of the video frame representing movement relative to a different video frame in the video sequence;

detecting region of interest (ROI) features within a combined foreground region of the video frame corresponding to the moving foreground region and a foreground region of the different video frame; and

approximating a shape of an ROI object within the video frame based on the detected ROI features.

2. The method of claim 1, further comprising, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, verifying the detected ROI features to select correct ROI features for the ROI object.

3. The method of claim 1, further comprising generating an output image that identifies the ROI object, and preferentially encoding the ROI object relative to non-ROI regions of the video frame.

4. The method of claim 1, wherein the different video frame is a previous video frame in the video sequence.

5. The method of claim 1, wherein identifying a moving foreground region comprises:

comparing first locations of pixels within the ROI object of the video frame to second locations of the pixels within the different video frame;

classifying pixels that have not moved from the second location as background pixels;

classifying pixels that have moved from the second location as foreground pixels; and

identifying the moving foreground region based on the foreground pixels.

6. The method of claim 1, wherein identifying a moving foreground region comprises performing a background subtraction process within the video frame to determine which regions are background regions based on motion information between the video frame and the different video frame of the video sequence.

7. The method of claim 1, wherein detecting ROI features comprises:

detecting a face mask within the combined foreground region based on chrominance values of skin-color regions within the video frame;

detecting eye features within the combined foreground region based on both chrominance values and luminance values of pixels within the detected face mask; and

detecting mouth features within the combined foreground region based on chrominance values of pixels within the detected face mask.

8. The method of claim 1, wherein the detected ROI features comprise detected facial feature candidates within the combined foreground region, the method further comprising verifying the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

9. The method of claim 1, wherein the combined foreground region includes more than one ROI object, the method further comprising, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, separating the detected ROI features into groups for the individual ROI objects, and applying a maximum matching scheme problem to the detected ROI features within the moving foreground region.

10. A computer-readable medium comprising instructions that cause a programmable processor to:

receive a video frame of a video sequence;

identify a moving foreground region of the video frame representing movement relative to a different video frame in the video sequence;

detect region of interest (ROI) features within a combined foreground region of the video frame corresponding to the moving foreground region and a foreground region of the different video frame; and

approximate a shape of an ROI object within the video frame based on the detected ROI features.

11. The computer-readable medium of claim 10, further comprising instructions that cause the programmable processor to, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, verify the detected ROI features to select correct ROI features for the ROI object.

12. The computer-readable medium of claim 10, wherein the instructions cause the processor to generate an output image that identifies the ROI object, and preferentially encoding the ROI object relative to non-ROI regions of the video frame.

13. The computer-readable medium of claim 10, wherein the different video frame is a previous video frame in the video sequence.

14. The computer-readable medium of claim 10, wherein the instructions cause the processor to:

compare a first locations of pixels within the ROI object of the video frame to a second locations of the pixels within the different video frame;

classify pixels that have not moved from the second location as background pixels;

classify pixels that have moved from the second location as foreground pixels; and

identify the moving foreground region based on the foreground pixels.

15. The computer-readable medium of claim 10, wherein the instructions cause the processor to perform a background subtraction process within the video frame to determine which regions are background regions based on motion information between the video frame and the different video frame of the video sequence, and identify the moving foreground region based on the background subtraction process.

16. The computer-readable medium of claim 10, wherein, for detection of ROI features, the instructions cause the processor to:

detect a face mask within the combined foreground region based on chrominance values of skin-color regions within the video frame;

detect eye features within the combined foreground region based on both chrominance values and luminance values of pixels within the detected face mask; and

detect mouth features within the combined foreground region based on chrominance values of pixels within the detected face mask.

17. The computer-readable medium of claim 10, wherein the detected ROI features comprise detected facial feature candidates within the combined foreground region, and the instructions cause the processor to verify the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

18. The computer-readable medium of claim 10, wherein the combined foreground region includes more than one ROI object, the method further comprising, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, separating the detected ROI features into groups for the individual ROI objects, and applying a maximum matching scheme problem to the detected ROI features within the moving foreground region.

19. A system comprising a processor programmed to:

receive a video frame in a video sequence;

identify a moving foreground region of the video frame representing movement relative to a different video frame in the video sequence;

detect region of interest (ROI) features within a combined foreground region of the video frame corresponding to the moving foreground region and a foreground region of the different video frame; and

approximate a shape of an ROI object within the video frame based on the detected ROI features.

20. The system of claim 19, wherein the processor, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, verifies the detected ROI features to select correct ROI features for the ROI object.

21. The system of claim 19, wherein the processor generates an output image that identifies the ROI object, and preferentially encoding the ROI object relative to non-ROI regions of the video frame.

22. The system of claim 19, wherein the different video frame is a previous video frame in the video sequence.

23. The system of claim 19, the processor:

compares a first locations of pixels within the ROI object of the video frame to a second locations of the pixels within the different video frame;

classifies pixels that have not moved from the second location as background pixels;

classifies pixels that have moved from the second location as foreground pixels; and

identifies the moving foreground region based on the foreground pixels.

24. The system of claim 19, wherein the processor performs a background subtraction process within the video frame to determine which regions are background regions based on motion information between the video frame and the different video frame of the video sequence, and identifies the moving foreground region based on the background subtraction process.

25. The system of claim 19, wherein, for detection of ROI features, the processor:

detects a face mask within the combined foreground region based on chrominance values of skin-color regions within the video frame;

detects eye features within the combined foreground region based on both chrominance values and luminance values of pixels within the detected face mask; and

detects mouth features within the combined foreground region based on chrominance values of pixels within the detected face mask.

26. The system of claim 19, wherein the detected ROI features comprise detected facial feature candidates within the combined foreground region, and the processor verifies the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

27. The system of claim 19, wherein the combined foreground region includes more than one ROI object, and the processor, if locations of the detected ROI features in the video frame do not substantially correspond to locations of other ROI features detected in the different video frame, separates the detected ROI features into groups for the individual ROI objects, and applies a maximum matching scheme problem to the detected ROI features within the moving foreground region.