Video frame motion-based automatic region-of-interest detection

• US 20070076957A1
• Filed: 02/28/2006
• Published: 04/05/2007
• Est. Priority Date: 10/05/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc., Zxibix Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wang, Haohong, Jiang, Xiaoyun, Quan, Shuxue, El-Maleh, Khaled, Chiu, Chlachuan
• Granted Patent: US 8,019,170 B2
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

The disclosure is directed to techniques for region-of-interest (ROI) video processing based on low-complexity automatic ROI detection within video frames of video sequences. The low-complexity automatic ROI detection may be based on characteristics of video sensors within video communication devices. In other cases, the low-complexity automatic ROI detection may be based on motion information for a video frame and a different video frame of the video sequence. The disclosed techniques include a video processing technique capable of tuning and enhancing video sensor calibration, camera processing, ROI detection, and ROI video processing within a video communication device based on characteristics of a specific video sensor. The disclosed techniques also include a sensor-based ROI detection technique that uses video sensor statistics and camera processing side-information to improve ROI detection accuracy. The disclosed techniques also include a motion-based ROI detection technique that uses motion information obtained during motion estimation in video processing.

Claims

1. A method comprising:

receiving a skin map of a video frame of a video sequence;

receiving motion information for the video frame and a different video frame of the video sequence; and

automatically detecting a region of interest (ROI) within the video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and a ROI within the different video frame.

2. The method of claim 1, wherein automatically detecting the ROI within the video frame comprises:

comparing a first macroblock of the video frame with a second macroblock of the different video frame corresponding to the first macroblock; and

selecting the first macroblock as a portion of the ROI within the video frame when the second macroblock sufficiently overlaps a ROI within the different video frame and the first macroblock sufficiently overlaps the skin map of the video frame.

3. The method of claim 2, further comprising dropping the first macroblock from consideration as a portion of the ROI within the video frame when at least one of the second macroblock does not at sufficiently overlap a ROI within the different video frame or the first macroblock does not sufficiently overlap the skin map of the video frame.

4. The method of claim 1, wherein receiving motion information comprises receiving motion vectors obtained by tracking motion of the ROI between the video frame and the different video frame of the video sequence.

5. The method of claim 1, further comprising:

receiving a skin map of another video frame of the video sequence; and

automatically detecting a ROI within the another video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and without reference to motion information for the another video frame and a different video frame of the video sequence

6. The method of claim 5, wherein automatically detecting the ROI within the another video frame comprises:

dividing the skin map into disconnected regions;

selecting candidate regions from the disconnected regions that include a largest area of the another video frame;

detecting ROI features within the candidate regions;

selecting one or more ROI regions from the candidate regions that include at least one of a largest number of ROI features or the largest area of the another video frame;

selecting ROI macroblocks from macroblocks of the another video frame that at least partially overlap with the one or more ROI regions within the another video frame; and

generating the ROI within the another video frame based on the selected ROI macroblocks.

7. The method of claim 6, further comprising verifying the detected ROI features within the video frame to select correct features for the ROI and remove false features from the set of ROI feature candidates.

8. The method of claim 6, further comprising performing morphological operations on the one or more ROI regions to close internal holes in the skin map of the video frame from the detected ROI features.

9. The method of claim 1, further comprising:

receiving the video frame of the video sequence from a video sensor;

generating sensor statistics for the video sensor;

detecting skin regions within the video frame based on the sensor statistics; and

generating the skin map of the video frame based on the detected skin regions.

10. The method of claim 1, wherein receiving a skin map comprises receiving a skin map of the video frame generated based on sensor statistics for a video sensor.

11. The method of claim 1, further comprising processing the video frame including the automatically detected ROI, wherein processing the video frame comprises preferentially processing the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

12. The method of claim 1, further comprising encoding the video frame including the automatically detected ROI, wherein encoding the video frame comprises preferentially encoding the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

13. The method of claim 1, further comprising:

selecting an automatic ROI detection mode from at least a first ROI detection mode and a second ROI detection mode;

automatically detecting a ROI from the video frame without reference to motion information for the video frame and a different video frame of the video sequence when the first ROI detection mode is selected; and

automatically detecting a ROI from the video frame based on motion information for the video frame and a different video frame of the video sequence when the second segmentation mode is selected.

14. The method of claim 13, wherein selecting an automatic ROI detection mode comprises determining an amount of motion activity between the video frame and the different video frame of the video sequence, and selecting the first ROI detection mode when the amount of motion activity is above a pre-determined level.

15. The method of claim 13, wherein selecting an automatic ROI detection mode comprises determining a number of consecutive video frames of the video sequence in which a ROI was automatically detected in the second ROI detection mode, and selecting the first ROI detection mode when the number of consecutive second ROI detection mode video frames is above a pre-determined level.

16. A computer readable medium comprising instructions that cause the programmable processor to:

receive a skin map of a video frame of a video sequence;

receive motion information for the video frame and a different video frame of the video sequence; and

automatically detect a region of interest (ROI) within the video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and a ROI within the different video frame.

17. The computer-readable medium of claim 16, wherein the instructions cause the programmable processor to:

compare a first macroblock of the video frame with a second macroblock of the different video frame corresponding to the first macroblock; and

select the first macroblock as a portion of the ROI within the video frame when the second macroblock sufficiently overlaps a ROI within the different video frame and the first macroblock sufficiently overlaps the skin map of the video frame.

18. The computer-readable medium of claim 17, further comprising instructions that cause the programmable processor to drop the first macroblock from consideration as a portion of the ROI within the video frame when at least one of the second macroblock does not at sufficiently overlap a ROI within the different video frame or the first macroblock does not sufficiently overlap the skin map of the video frame.

19. The computer-readable medium of claim 16, wherein the instructions cause the programmable processor to receive motion vectors obtained by tracking motion of the ROI between the video frame and the different video frame of the video sequence.

20. The computer-readable medium of claim 16, further comprising instructions that cause the programmable processor to:

receive a skin map of another video frame of the video sequence; and

automatically detect a ROI within the another video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and without reference to motion information for the another video frame and a different video frame of the video sequence

21. The computer-readable medium of claim 20, wherein the instructions cause the programmable processor to:

divide the skin map into disconnected regions;

select candidate regions from the disconnected regions that include a largest area of the another video frame;

detect ROI features within the candidate regions;

select one or more ROI regions from the candidate regions that include at least one of a largest number of ROI features or the largest area of the another video frame;

select ROI macroblocks from macroblocks of the another video frame that at least partially overlap with the one or more ROI regions within the another video frame; and

generate the ROI within the another video frame based on the selected ROI macroblocks.

22. The computer-readable medium of claim 21, further comprising instructions that cause the programmable processor to verify the detected ROI features within the video frame to select correct features for the ROI and remove false features from the set of ROI feature candidates.

23. The computer-readable medium of claim 21, further comprising instructions that cause the programmable processor to perform morphological operations on the one or more ROI regions to close internal holes in the skin map of the video frame from the detected ROI features.

24. The computer-readable medium of claim 16, further comprising instructions that cause the programmable processor to:

receive the video frame of the video sequence from a video sensor;

generate sensor statistics for the video sensor;

detect skin regions within the video frame based on the sensor statistics; and

generate the skin map of the video frame based on the detected skin regions.

25. The computer-readable medium of claim 16, wherein the instructions cause the programmable processor to receive a skin map of the video frame generated based on sensor statistics for a video sensor.

26. The computer-readable medium of claim 16, further comprising instructions that cause the programmable processor to process the video frame including the automatically detected ROI, wherein the instructions cause the programmable processor to preferentially process the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

27. The computer-readable medium of claim 16, further comprising instructions that cause the programmable processor to encode the video frame including the automatically detected ROI, wherein the instructions cause the programmable processor to preferentially encode the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

28. The computer-readable medium of claim 16, further comprising instructions that cause the programmable processor to:

select an automatic ROI detection mode from at least a first ROI detection mode and a second ROI detection mode;

automatically detect a ROI from the video frame without reference to motion information for the video frame and a different video frame of the video sequence when the first ROI detection mode is selected; and

automatically detect a ROI from the video frame based on motion information for the video frame and a different video frame of the video sequence when the second segmentation mode is selected.

29. The computer-readable medium of claim 28, wherein the instructions cause the programmable processor to determine an amount of motion activity between the video frame and the different video frame of the video sequence, and select the first ROI detection mode when the amount of motion activity is above a pre-determined level.

30. The computer-readable medium of claim 28, wherein the instructions cause the programmable processor to determine a number of consecutive video frames of the video sequence in which a ROI was automatically detected in the second ROI detection mode, and select the first ROI detection mode when the number of consecutive second ROI detection mode video frames is above a pre-determined level.

31. A video processing system comprising:

a skin region detector that generates a skin map of a video frame of a video sequence;

a region of interest (ROI) video processing module that generates motion information for the video frame and a different video frame of the video sequence; and

a ROI detector that receives the skin map and the motion information for the video frame and automatically detects the ROI within the video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and a ROI within the different video frame.

32. The system of claim 31, wherein the ROI detector:

compares a first macroblock of the video frame with a second macroblock of the different video frame corresponding to the first macroblock; and

selects the first macroblock as a portion of the ROI within the video frame when the second macroblock sufficiently overlaps a ROI within the different video frame and the first macroblock sufficiently overlaps the skin map of the video frame.

33. The system of claim 32, wherein the ROI detector drops the first macroblock from consideration as a portion of the ROI within the video frame when at least one of the second macroblock does not at sufficiently overlap a ROI within the different video frame or the first macroblock does not sufficiently overlap the skin map of the video frame.

34. The system of claim 32, wherein the second macroblock sufficiently overlaps the ROI within the different video frame when more than a predetermined percentage of the area of the second macroblock overlaps the ROI of the previous video frame.

35. The system of claim 32, wherein the first macroblock sufficiently overlaps the skin map of the video frame when more than a predetermined percentage of the area of the first macroblock overlaps the skin map of the video frame.

36. The system of claim 31, wherein the ROI video processing module tracks motion of the ROI between the video frame and the different video frame of the video sequence to generate motion vectors.

37. The system of claim 31,

wherein the skin region detector generates a skin map of another video frame of the video sequence;

and

wherein the ROI detector receives the skin map of the another video frame and automatically detects a ROI within the another video frame based on locations of macroblocks in the video frame relative to the skin map of the video frame and without reference to motion information for the another video frame and a different video frame of the video sequence

38. The system of claim 37, wherein the ROI detector includes:

a region labeling module that divides the skin map into disconnected regions;

a region selection module that selects candidate regions from the disconnected regions that include a largest area of the another video frame;

a feature detection and verification module that detects ROI features within the candidate regions;

a ROI region selection module that selects one or more ROI regions from the candidate regions that include at least one of a largest number of ROI features or the largest area of the another video frame; and

a ROI macroblock selection module that selects ROI macroblocks from macroblocks of the another video frame that at least partially overlap with the one or more ROI regions within the another video frame,

wherein the ROI detector generates the ROI within the another video frame based on the selected ROI macroblocks.

39. The system of claim 38, wherein the feature detection and verification module verifies the detected ROI features within the video frame to select correct features for the ROI and remove false features from the set of ROI feature candidates.

40. The system of claim 38, further comprising a morphological operation module that performs morphological operations on the one or more ROI regions to close internal holes in the skin map of the video frame from the detected ROI features.

41. The system of claim 31, further comprising:

a camera processing module that receives the video frame of the video sequence from a video sensor; and

a sensor calibration module that generates sensor statistics for the video sensor,

wherein the skin region detector detects skin regions within the video frame based on the sensor statistics and generates the skin map of the video frame based on the detected skin regions.

42. The system of claim 31, wherein the ROI detector receives a skin map generated based on sensor statistics for a video sensor.

43. The system of claim 31, wherein the ROI video processing module processes the video frame including the automatically detected ROI, wherein the ROI video processing module preferentially processes the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

44. The system of claim 31, wherein the ROI video processing module comprises a ROI video coding module that encodes the video frame including the automatically detected ROI, wherein the ROI video coding module preferentially encodes the automatically detected ROI within the video frame relative to non-ROI areas of the video frame.

45. The system of claim 31, further comprising a ROI detection controller that:

selects an automatic ROI detection mode from at least a first ROI detection mode and a second ROI detection mode;

decides to automatically detect a ROI from the video frame without reference to motion information for the video frame and a different video frame of the video sequence when the first ROI detection mode is selected; and

decides to automatically detect a ROI from the video frame based on motion information for the video frame and a different video frame of the video sequence when the second segmentation mode is selected.

46. The system of claim 45, wherein the ROI detection controller determines an amount of motion activity between the video frame and the different video frame of the video sequence, and selects the first ROI detection mode when the amount of motion activity is above a pre-determined level.

47. The system of claim 45, wherein the ROI detection controller determines a number of consecutive video frames of the video sequence in which a ROI was automatically detected in the second ROI detection mode, and selects the first ROI detection mode when the number of consecutive second ROI detection mode video frames is above a pre-determined level.