Intra-mode region-of-interest video object segmentation

• US 8,265,349 B2
• Filed: 02/07/2006
• Issued: 09/11/2012
• Est. Priority Date: 02/07/2006
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wang, Haohong, El-Maleh, Khaled Helmi
• Primary Examiner(s): Liew, Alex
• Application Number:
  US11/350,376
• Publication Number:
  US 20070183663A1
• Time in Patent Office: 2,408 Days
• Field of Search: 382115-118, 382173-180
• US Class Current: 382/118
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object

Abstract

The disclosure is directed to techniques for automatic segmentation of a region-of-interest (ROI) video object from a video sequence. ROI object segmentation enables selected ROI or “foreground” objects of a video sequence that may be of interest to a viewer to be extracted from non-ROI or “background” areas of the video sequence. Examples of a ROI object are a human face or a head and shoulder area of a human body. The disclosed techniques include a hybrid technique that combines ROI feature detection, region segmentation, and background subtraction. In this way, the disclosed techniques may provide accurate foreground object generation and low-complexity extraction of the foreground object from the video sequence. A ROI object segmentation system may implement the techniques described herein. In addition, ROI object segmentation may be useful in a wide range of multimedia applications that utilize video sequences, such as video telephony applications and video surveillance applications.

Claims

1. A method performed by an automatic region of interest (ROI) object segmentation apparatus, the method comprising:

receiving, in the automatic ROI object segmentation apparatus, a video frame of a video sequence;

detecting, in the automatic ROI object segmentation apparatus, region of interest (ROI) features within the video frame independent of other video frames of the video sequence;

verifying the detected ROI features within the video frame based on a geometric relationship of the detected ROI features;

approximating, in the automatic ROI object segmentation apparatus, a ROI object shape within the video frame based on the detected ROI features and the geometric relationship of the detected ROI features,

wherein approximating a ROI object shape comprises approximating the ROI object shape based on a geometric model defined by a location of the detected ROI features;

segmenting, in the automatic ROI object segmentation apparatus, the video frame into multiple candidate regions, wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

selecting, in the automatic ROI object segmentation apparatus, one or more of the candidate regions located within the approximated ROI object shape of the video frame as an ROI object.

2. The method of claim 1, further comprising generating an output image that identifies the ROI object, and preferentially encoding the ROI object relative to non-ROI regions of the video frame.

3. The method of claim 1, wherein selecting one or more candidate regions comprises selecting one or more of the candidate regions as the ROI object without reference to motion information for the video frame in the video sequence.

4. The method of claim 1, wherein detecting ROI features comprises:

detecting a face mask within the video frame based on chrominance values of skin-color regions within the video frame;

detecting eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask; and

detecting mouth features within the video frame based on chrominance values of pixels within the detected face mask.

5. The method of claim 1, wherein the detected ROI features comprise detected facial feature candidates within the video frame, and wherein verifying the detected ROI features further comprises verifying the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

6. The method of claim 5, wherein verifying the detected facial feature candidates comprises:

detecting valley regions within the video frame;

comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

7. The method of claim 5, wherein verifying the detected facial feature candidates comprises:

comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

8. The method of claim 5, wherein verifying the detected facial feature candidates comprises:

combining eye feature candidates and mouth feature candidates into eye-mouth triangles; and

verifying the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

9. The method of claim 1, wherein the video frame includes more than one ROI object, the method further comprising separating the detected ROI features into groups for the individual ROI objects, and applying maximum matching graph theory to the detected ROI features within the video frame.

10. The method of claim 1, wherein approximating a ROI object shape comprises approximating the ROI object shape based on a geometric model defined by a location of the detected ROI features, wherein the approximate ROI object shape comprises a human head and shoulder object shape within the video frame.

11. The method of claim 1, wherein segmenting the video frame comprises:

classifying each pixel within the video frame based on similarity of intensity to neighboring pixels; and

merging the similar pixels into a minimum number of the candidate regions within the video frame.

12. The method of claim 1, wherein selecting one or more of the candidate regions comprises selecting one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, the method further comprising generating a foreground object by merging the selected one or more of the candidate regions.

13. The method of claim 12, wherein the video frame includes more than one foreground object, the method further comprising merging the more than one foreground objects to form an output image and sending the output image to a multimedia application.

14. A non-transitory computer-readable medium having thereon instructions that cause a programmable processor to:

receive a video frame of a video sequence;

detect region of interest (ROI) features within the video frame independent of other video frames of the video sequence;

verify the detected ROI features within the video frame based on a geometric relationship of the detected ROI features;

approximate a ROI object shape within the video frame based on the detected ROI features,

wherein the ROI object shape is approximated based on a geometric model defined by a location of the detected ROI features and the geometric relationship of the detected ROI features;

segment the video frame into multiple candidate wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

select one or more of the candidate regions located within the approximated ROI object shape of the video frame as an ROI object.

15. The non-transitory computer readable medium of claim 14, wherein the instructions cause the processor to generate an output image that identifies the ROI object, and preferentially encode the ROI object relative to non-ROI regions of the video frame.

16. The non-transitory computer readable medium of claim 14, further comprising instructions to cause the processor to:

select one or more of the candidate regions as the ROI object without reference to motion information for the video frame and a different video frame in the video sequence.

17. The non-transitory computer readable medium of claim 14, wherein the instructions that cause the programmable processor to detect ROI features cause the programmable processor to:

detect a face mask within the video frame based on chrominance values of skin-color regions within the video frame;

detect eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask; and

detect mouth features within the video frame based on chrominance values of pixels within the detected face mask.

18. The non-transitory computer readable medium of claim 14, wherein the detected ROI features comprise detected facial feature candidates within the video frame, further comprising instructions that cause the programmable processor to verify the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

19. The non-transitory computer readable medium of claim 18, wherein the instructions cause the programmable processor to:

detect valley regions within the video frame;

compare locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

remove a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

20. The non-transitory computer readable medium of claim 18, wherein the instructions cause the programmable processor to:

compare each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assign a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

remove a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

21. The non-transitory computer readable medium of claim 18, wherein the instructions cause the programmable processor to:

combine eye feature candidates and mouth feature candidates into eye-mouth triangles; and

verify the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

22. The non-transitory computer readable medium of claim 14, wherein the video frame includes more than one ROI object, further comprising instructions that cause the programmable processor to separate the detected ROI features into groups for the individual ROI objects, and apply maximum matching graph theory to the detected ROI features within the video frame.

23. The non-transitory computer readable medium of claim 14, wherein the instructions cause the programmable processor to approximate the ROI object shape based on a geometric model defined by a location of the detected ROI features.

24. The non-transitory computer readable medium of claim 14, wherein the instructions that cause the programmable processor to segment the video frame cause the programmable processor to:

classify each pixel within the video frame based on similarity of intensity to neighboring pixels; and

merge the similar pixels into a minimum number of candidate regions within the first frame.

25. The non-transitory computer readable medium of claim 17, wherein the instructions cause the programmable processor to select one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, further comprising instructions that cause the programmable processor to generate a foreground object by merging the selected one or more of the candidate regions.

26. The non-transitory computer readable medium of claim 25, wherein the video frame includes more than one foreground object, further comprising instructions that cause the programmable processor to merge the more than one foreground objects to form the output image and send the output image to a multimedia application.

27. An automatic region of interest (ROI) object segmentation system comprising:

a ROI feature detector that detects ROI features within a video frame of a video sequence independent of other video frames of the video sequence;

a feature verification module that verifies the detected ROI features within the video frame based on a geometric relationship of the detected ROI features;

an object shape approximation module that approximates a ROI object shape within the video frame based on the detected ROI features and the geometric relationship of the detected ROI features,

wherein the ROI object shape is approximated based on a geometric model defined by a location of the detected ROI features;

a region segmentation module that segments the video frame into multiple candidate regions, wherein segmentation by the region segmentation module comprises classifying each pixels within the video frame based on a similarity to neighboring pixels; and

an object generation module that selects one or more of the candidate regions located within the approximated ROI object shape of the video frame as an ROI object.

28. The system of claim 27, further comprising:

an object fusion module that generates an output image that identifies the ROI object; and

an encoder that preferentially encodes the ROI object relative to non-ROI regions of the video frame.

29. The system of claim 27, wherein the object generation module selects the one or more of the candidate regions as the ROI object without reference to motion information for the video frame in the video sequence.

30. The system of claim 27, wherein the ROI feature detector includes a face mask detector that detects a face mask within the video frame based on chrominance values of skin-color regions within the video frame, an eye detector that detects eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask, and a mouth detector that detects mouth features within the video frame based on chrominance values of pixels within the detected face mask.

31. The system of claim 27, wherein the detected ROI features comprise detected facial feature candidates within the video frame, wherein the feature verification module verifies the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates, and wherein the feature verification module:

detects valley regions within the video frame;

compares locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

removes a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

32. The system of claim 31, wherein the feature verification module:

compares each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assigns a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

removes a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

33. The system of claim 31, wherein the feature verification module:

combines eye feature candidates and mouth feature candidates into eye-mouth triangles; and

verifies the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

34. The system of claim 27, wherein the video frame includes more than one ROI object, further comprising a multi-face separation module that separates the detected ROI features into groups for the individual ROI objects, and applies maximum matching graph theory to the detected ROI features within the video frame to separate the detected ROI features.

35. The system of claim 27, wherein the object shape approximation module approximates the ROI object shape based on a geometric model defined by a location of the detected ROI features, wherein the approximate ROI object shape comprises a human head and shoulder object shape within the video frame.

36. The system of claim 27, wherein the region segmentation module:

classifies each pixel within the video frame based on similarity of intensity to neighboring pixels; and

merges the similar pixels to form a minimum number of candidate regions within the first frame.

37. The system of claim 27, further comprising an object generation module that selects one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, wherein the object generation module generates a foreground object by merging the selected one or more of the candidate regions.

38. The system of claim 37, wherein the video frame includes more than one foreground objects, wherein the object fusing module merges the more than one foreground objects to form the output image and sends the output image to a multimedia application.

39. A method performed by an automatic region of interest (ROI) object segmentation apparatus, the method comprising:

receiving, in the automatic ROI object segmentation apparatus, a video frame of a video sequence;

detecting, in the automatic ROI object segmentation apparatus, region of interest (ROI) features within the video frame independent of other video frames of the video sequence, wherein the ROI features include a face mask including eye features and mouth features;

combining, in the automatic ROI object segmentation apparatus, eye feature candidates and mouth feature candidates into eye-mouth triangles;

verifying the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and

approximating an ROI object shape within the video frame based on the detected ROI features, wherein the ROI object shape is approximated based on a geometric model defined by a location of the detected ROI features; and

segmenting, in the automatic ROI object segmentation apparatus, the video frame into multiple candidate regions, wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

selecting, in the automatic ROI object segmentation apparatus, one or more of the candidate regions located within the approximated ROI object shape as an ROI object.

40. The method of claim 39, wherein detecting ROI features comprises detecting the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

41. The method of claim 39, wherein verifying the detected facial feature candidates comprises:

detecting valley regions within the video frame;

comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

42. The method of claim 39, wherein verifying the detected facial feature candidates comprises:

comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

43. A non-transitory computer-readable medium having thereon instructions that cause a programmable processor to:

receive a video frame of a video sequence;

detect region of interest (ROI) features within the video frame independent of other video frames of the video sequence, wherein the ROI features include a face mask including eye features and mouth features;

combine eye feature candidates and mouth feature candidates into eye-mouth triangles;

verify the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and

approximate an ROI object shape within the video frame based on the detected ROI features, wherein the ROI object shape is approximated based on a geometric model defined by a location of the detected ROI features; and

segment the video frame into multiple candidate regions, wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

select one or more of the candidate regions located within the approximated ROI object shape as an ROI object.

44. The non-transitory computer readable medium of claim 43, wherein the instructions cause the processor to detect the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

45. The non-transitory computer readable medium of claim 43, wherein verification of the detected facial feature candidates includes:

detecting valley regions within the video frame;

comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

46. The non-transitory computer readable medium of claim 43, wherein verification of the detected facial feature candidates comprises:

comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

47. A system comprising:

a region of interest (ROI) feature detector that detects ROI features within a video frame of a video sequence independent of other video frames of the video sequence, wherein the ROI features include a face mask including eye features and mouth features;

a feature verification module that combines eye feature candidates and mouth feature candidates into eye-mouth triangles, and verifies the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame;

an object shape approximation module that approximates a ROI object shape within the video frame based on a geometric model defined by a location of the detected ROI features;

a region segmentation module that segments the video frame into multiple candidate regions, wherein segmenting the frame into multiple candidate regions comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

an object generation module that selects one or more of the candidate regions located within the approximated ROI object shape as an ROI object.

48. The system of claim 47, wherein the ROI feature detector detects the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

49. The system of claim 47, wherein the feature verification module:

detects valley regions within the video frame;

compares locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;

removes a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

50. The system of claim 49, wherein the feature verification module:

compares each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;

assigns a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and

removes a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

51. An apparatus for automatically segmenting a region of interest (ROI), the apparatus comprising:

means for receiving, in the automatic ROI object segmentation apparatus, a video frame of a video sequence;

means for detecting, in the automatic ROI object segmentation apparatus, region of interest (ROI) features within the video frame independent of other video frames of the video sequence;

means for verifying the detected ROI features within the video frame based on a geometric relationship of the detected ROI features;

means for approximating, in the automatic ROI object segmentation apparatus, a ROI object shape within the video frame based on the detected ROI features and the geometric relationship of the detected ROI features,

wherein approximating a ROI object shape comprises approximating the ROI object shape based on a geometric model defined by a location of the detected ROI features;

means for segmenting, in the automatic ROI object segmentation apparatus, the video frame into multiple candidate regions, wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

means for selecting, in the automatic ROI object segmentation apparatus, one or more of the candidate regions located within the approximated ROI object shape of the video frame as an ROI object.

52. An apparatus for automatically segmenting a region of interest (ROI), the apparatus comprising:

means for receiving, in the automatic ROI object segmentation apparatus, a video frame of a video sequence;

means for detecting, in the automatic ROI object segmentation apparatus, region of interest (ROI) features within the video frame independent of other video frames of the video sequence, wherein the ROI features include a face mask including eye features and mouth features;

means for combining, in the automatic ROI object segmentation apparatus, eye feature candidates and mouth feature candidates into eye-mouth triangles;

means for verifying the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and

means for approximating an ROI object shape within the video frame based on the detected ROI features, wherein the ROI object shape is approximated based on a geometric model defined by a location of the detected ROI features; and

means for segmenting, in the automatic ROI object segmentation apparatus, the video frame into multiple candidate regions, wherein segmenting comprises classifying pixels within the video frame based on a similarity to neighboring pixels; and

means for selecting, in the automatic ROI object segmentation apparatus, one or more of the candidate regions located within the approximated ROI object shape as an ROI object.