Segmenting video input using high-level feedback

• US 6,999,620 B1
• Filed: 12/10/2001
• Issued: 02/14/2006
• Est. Priority Date: 12/10/2001
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Harville, Michael
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Hung, Yubin
• Application Number:
  US10/006,687
• Time in Patent Office: 1,527 Days
• Field of Search: 382/103, 382/106, 382156-159, 382/164, 382/173, 382/176, 382/180, 382224-228, 358/464, 706/20
• US Class Current: 382/173
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Systems and methods of segmenting video input are described. These systems and methods use feedback processing that operates on levels of information above the pixel level, such as image regions, frame-wide statistics, or object and event semantics. In one aspect, one or more pixel-level historical models of spatially local pixel observations is maintained. Pixels are segmented into two or more labeled groups based at least in part upon comparison of pixel-level video input with the one or more pixel-level historical models. The pixel-level historical models are updated based at least in part upon feedback maps identifying pixels respectively segmented into the one or more labeled groups in conformity with a spatially non-local segmentation model.

Claims

1. A method of segmenting video input characterized by a time series of video frames of observable pixel data, comprising:

maintaining one or more pixel-level historical models of spatially local pixel observations;

segmenting pixels into two or more labeled groups based at least in part upon comparison of pixel-level video input with the one or more pixel-level historical models; and

updating the one or more pixel-level historical models based at least in part upon one or more feedback maps identifying pixels respectively segmented into the one or more labeled groups in conformity with a spatially non-local segmentation model.

2. The method of claim 1, wherein a history of pixel observations at each pixel is modeled by a mixture of Gaussian distributions.

3. The method of claim 1, wherein pixels are segmented into a background group and a foreground group.

4. The method of claim 1, wherein a feedback map identifies pixels segmented correctly according to a spatially non-local segmentation model.

5. The method of claim 4, wherein the spatially non-local segmentation model defines spatially non-local observation characteristics of pixels belonging to one of the labeled groups.

6. The method of claim 4, wherein a pixel-level historical model is not updated at pixels identified as being correctly segmented.

7. The method of claim 1, wherein a feedback map identifies pixels segmented incorrectly according to a spatially non-local segmentation model.

8. The method of claim 7, wherein the spatially non-local segmentation model defines spatially non-local observation characteristics of pixels that should have been excluded from one of the labeled groups.

9. The method of claim 7, wherein a pixel-level historical model is updated at pixels identified as being incorrectly segmented.

10. The method of claim 9, wherein updating the pixel-level historical model at pixels identified as being incorrectly segmented comprises maintaining a per-pixel inclusion error model of pixel observations associated with occurrences of incorrect segmentation labeling.

11. The method of claim 10, wherein components of the per-pixel inclusion error model corresponding to individual pixels include respective mixtures of Gaussian distributions.

12. The method of claim 11, wherein updating a pixel-level historical model comprises merging a per-pixel historical model and a per-pixel inclusion error model.

13. The method of claim 1, further comprising generating the feedback maps based at least in part upon analysis of spatially non-local video frame features.

14. The method of claim 13, wherein the feedback maps are generated based at least in part upon one or more of an image region analysis, a frame-wide image statistics analysis, or an analysis of the object or event content of the video frames.

15. The method of claim 14, wherein one or more of the feedback maps are generated based at least in part upon depth information or stereo disparity information, or both.

16. The method of claim 1, further comprising generating one or more confidence maps associating pixels with respective measures of segmentation accuracy.

17. The method of claim 16, further comprising merging multiple confidence maps to produce a merged confidence map.

18. The method of claim 17, wherein the measures of segmentation accuracy are real numbers, and the step of merging multiple confidence maps comprises adding the multiple segmentation accuracy measures respectively associated with each pixel.

19. The method of claim 17, further comprising thresholding the merged confidence map to produce one or more feedback maps.

20. The method of claim 16, wherein each of the confidence maps is generated based at least in part upon one or more of an image region analysis, a frame-wide image statistics analysis, or an analysis of the object or event content of the video frames.

21. The method of claim 20, wherein a pixel-level historical model includes a mixture of Gaussian distributions of pixel observations.

22. The method of claim 1, wherein one or more pixel-level historical models incorporate per pixel depth information or stereo disparity information, or both.

23. The method of claim 1, wherein pixels are segmented based at least in part upon depth information or stereo disparity information, or both.

24. The method of claim 1, wherein one or more feedback maps are generated by one or more of a person detector and tracker module, a rapid illumination change detector module, a camera gain change detector module, or a sudden camera motion detector module.

25. A system for segmenting video input characterized by a time series of video frames of observable pixel data, comprising one or more processing modules operable to:

maintain one or more pixel-level historical models of spatially local pixel observations;

segment pixels into two or more labeled groups based at least in part upon comparison of pixel-level video input with the one or more pixel-level historical models; and

update the one or more pixel-level historical models based at least in part upon one or more feedback maps identifying pixels respectively segmented into the one or more labeled groups in conformity with a spatially non-local segmentation model.

26. The system of claim 25, wherein a history of pixel observations at each pixel is modeled by a mixture of Gaussian distributions.

27. The system of claim 25, wherein pixels are segmented into a background group and a foreground group.

28. The system of claim 25, wherein a feedback map identifies pixels segmented correctly according to a spatially non-local segmentation model.

29. The system of claim 28, wherein the spatially non-local segmentation model defines spatially non-local observation characteristics of pixels belonging to one of the labeled groups.

30. The system of claim 28, wherein a pixel-level historical model is not updated at pixels identified as being correctly segmented.

31. The system of claim 25, wherein a feedback map identifies pixels segmented incorrectly according to a spatially non-local segmentation model.

32. The system of claim 31, wherein the spatially non-local segmentation model defines spatially non-local observation characteristics of pixels that should have been excluded from one of the labeled groups.

33. The system of claim 31, wherein a pixel-level historical model is updated at pixels identified as being incorrectly segmented.

34. The system of claim 33, wherein the pixel-level historical model is updated at pixels identified as being incorrectly segmented by maintaining a per-pixel inclusion error model of pixel observations associated with occurrences of incorrect segmentation.

35. The system of claim 34, wherein components of the per-pixel inclusion error model corresponding to individual pixels includes respective mixtures of Gaussian distributions.

36. The system of claim 35, wherein a pixel-level historical model is updated by merging a per-pixel historical model and a per-pixel inclusion error model.

37. The system of claim 25, wherein one or more of the processing modules are operable to generate feedback maps based at least in part upon an analysis of spatially non-local video frame features.

38. The system of claim 37, wherein the feedback maps are generated based at least in part upon one or more of an image region analysis, a frame-wide image statistics analysis, or an analysis of the object or event content of the video frames.

39. The system of claim 25, wherein one or more of the processing modules are operable to generate one or more confidence maps associating pixels with respective measures of segmentation accuracy.

40. The system of claim 39, wherein one or more of the processing modules are operable to merge multiple confidence maps to produce a merged confidence map.

41. The system of claim 40, wherein the measures of segmentation accuracy are real numbers, and multiple confidence maps are merged by adding the multiple segmentation accuracy measures respectively associated with each pixel.

42. The system of claim 40, wherein one or more of the processing modules are operable to threshold the merged confidence map to produce one or more feedback maps.

43. The system of claim 39, wherein each of the confidence maps is generated based at least in part upon one or more of an image region analysis, a frame-wide image statistics analysis, or an analysis of the object or event content of the video frames.

44. The system of claim 43, wherein a pixel-level historical model includes a mixture of Gaussian distributions of pixel observations.

45. A computer program for segmenting video input characterized by a time series of video frames of observable pixel data, the computer program residing on a computer-readable medium and comprising computer-readable instructions for causing a computer to:

maintain one or more pixel-level historical models of spatially local pixel observations;

segment pixels into two or more labeled groups based at least in part upon comparison of pixel-level video input with the one or more pixel-level historical models; and

update the one or more pixel-level historical models based at least in part upon feedback maps identifying pixels respectively segmented into the one or more labeled groups in conformity with a spatially non-local segmentation model.