Systems and methods for accurate user foreground video extraction

• US 8,818,028 B2
• Filed: 04/08/2011
• Issued: 08/26/2014
• Est. Priority Date: 04/09/2010
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Meyer, Greg, Do, Minh N., Lin, Dennis, Patel, Sanjay J., Nguyen, Quang H.
• Primary Examiner(s): Cunningham, Gregory F
• Application Number:
  US13/083,470
• Publication Number:
  US 20110249190A1
• Time in Patent Office: 1,236 Days
• Field of Search: 348/169, 348/208.14, 382/103, 382/106
• US Class Current: 382/103
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A color image and a depth image of a live video are received. Each of the color image and the depth image are processed to identify a foreground, background, and an unknown region band of the live video. The unknown region band may comprise pixels between the foreground and the background. Further processing is performed to segment the pixels of the unknown region band between the foreground and the background. As such, processing is performed on the unknown region band in order to provide an improved user foreground video.

Claims

1. A method for processing a video, the method comprising:

receiving, using a computer, a depth image and a color image of the video from at least one camera;

identifying a categorization for each pixel of the depth image of the video based on a comparison of a depth value for each pixel and at least one pixel history, the categorization comprises at least one of a user pixel, background pixel, and unknown pixel;

creating a region map based on the categorization for each of the pixels;

identifying an unclear region of the color image based on the region map;

creating an unclear region band of the color image, the unclear region band comprises categorized unclear pixels of the color image; and

distributing the categorized unclear pixels of the unclear region band of the color image into a foreground region and a background region.

2. The method as set forth in claim 1, further comprising grouping each of the pixels of the depth image of the video, the grouping comprising comparing the categorization and depth value of each of the pixels with a categorization and depth value of at least one neighboring pixel, if the pixel and the neighboring pixel comprise the same categorization and a similar depth value, then the pixel and the neighboring pixel are grouped into a connected component.

3. The method as set forth in claim 2, further comprising detecting if a connected component is in motion between the depth image frame and a next depth image frame, the motion is detected by comparing a difference in the depth value of at least one pixel of the connected component in the depth image frame and a depth value of at least one pixel of the connected component in the next depth image frame, a connected component comprising user pixels is determined to be a user, a connected component comprising background pixels in motion is determined to be a user if a head of the user is detected, a connected component comprising unknown pixels is determined to be a user if a head of the user is detected.

4. The method as set forth in claim 3, wherein the region map comprises categorizing user pixels in the connected component as foreground pixels if a user associated with the connected component is enabled, the user pixels are categorized as background pixels if the user associated with the connected component is disabled.

5. The method as set forth in claim 3, further comprising detecting features and gestures of the user for a connected component comprising user pixels, the detection of features comprises locating the user'"'"'s head and hands, the detection of gestures comprises detecting a movement of the user'"'"'s hands.

6. The method as set forth in claim 1, further comprising a foreground region filling if a resolution of the depth image is less than a resolution of the color image, the foreground region filling comprising categorizing at least one unknown pixel as a foreground pixel if the unknown pixel is surrounded by foreground pixels.

7. The method as set forth in claim 1, wherein the unclear region band is created by expanding, on a user body, a contour line separating the foreground region and the background region on the user body outwards and inwards and expanding the unclear region band surrounding a head of the user.

8. The method as set forth in claim 1, wherein the distributing of the categorized unclear pixels of the unclear region band comprises performing a graph cut on unclear pixels in the unclear region and adjacent foreground and background pixels, the graph cut segments the unclear pixels into at least two subsets comprising a foreground region comprising foreground pixels and a background region comprising background pixels.

9. A system, comprising at least one processor and memory, for processing a video, the system comprising:

a module for receiving a depth image and a color image of the video from at least one camera;

a module for identifying a categorization for each pixel of the depth image of the video based on a comparison of a depth value for each pixel and at least one pixel history, the categorization comprises at least one of a user pixel, background pixel, and unknown pixel;

a module for creating a region map based on the categorization for each of the pixels;

a module for identifying an unclear region of the color image based on the region map;

a module for creating an unclear region band of the color image, the unclear region band comprises categorized unclear pixels of the color image; and

a module for distributing the categorized unclear pixels of the unclear region band of the color image into a foreground region and a background region.

10. The system as set forth in claim 9, further comprising grouping each of the pixels of the depth image of the video, the grouping comprising comparing the categorization and depth value of each of the pixels with a categorization and depth value of at least one neighboring pixel, if the pixel and the neighboring pixel comprise the same categorization and a similar depth value, then the pixel and the neighboring pixel are grouped into a connected component.

11. The system as set forth in claim 10, further comprising detecting if a connected component is in motion between the depth image frame and a next depth image frame, the motion is detected by comparing a difference in the depth value of at least one pixel of the connected component in the depth image frame and a depth value of at least one pixel of the connected component in the next depth image frame, a connected component comprising user pixels is determined to be a user, a connected component comprising background pixels in motion is determined to be a user if a head of the user is detected, a connected component comprising unknown pixels is determined to be a user if a head of the user is detected.

12. The system as set forth in claim 11, wherein the region map comprises categorizing user pixels in the connected component as foreground pixels if a user associated with the connected component in motion is enabled, the foreground pixels are categorized as background pixels if the user associated with the connected component is disabled.

13. The system as set forth in claim 11, further comprising detecting features and gestures of the user for a connected component comprising user pixels, the detection of features comprises locating the user'"'"'s head and hands, the detection of gestures comprises detecting a movement of the user'"'"'s hands.

14. The system as set forth in claim 9, further comprising a foreground region filling if a resolution of the depth image is less than a resolution of the color image, the foreground region filling comprising categorizing at least one unknown pixel as a foreground pixel if the unknown pixel is surrounded by foreground pixels.

15. The system as set forth in claim 9, wherein the unclear region band is created by expanding, on a user body, a contour line separating the foreground region and the background region on the user body outwards and inwards and expanding the unclear region band surrounding a head of the user.

16. The system as set forth in claim 9, wherein the distributing of the categorized unclear pixels of the unclear region band comprises performing a graph cut on unclear pixels in the unclear region and adjacent foreground and background pixels, the graph cut segments the unclear pixels into at least two subsets comprising a foreground region comprising foreground pixels and a background region comprising background pixels.

17. A non-transitory computer readable medium carrying one or more instructions for processing a video, wherein the one or more instructions, when executed by one or more processors, causes the one or more processors to perform the steps of:

receiving a depth image and a color image of the video from at least one camera;

identifying a categorization for each pixel of the depth image of the video based on a comparison of a depth value for each pixel and at least one pixel history, the categorization comprises at least one of a user pixel, background pixel, and unknown pixel;

creating a region map based on the categorization for each of the pixels;

identifying an unclear region of the color image based on the region map;

creating an unclear region band of the color image, the unclear region band comprises categorized unclear pixels of the color image; and

distributing the categorized unclear pixels of the unclear region band of the color image into a foreground region and a background region.

18. The non-transitory computer readable medium as set forth in claim 17, further comprising grouping each of the pixels of the depth image of the video, the grouping comprising comparing the categorization and depth value of each of the pixels with a categorization and depth value of at least one neighboring pixel, if the pixel and the neighboring pixel comprise the same categorization and a similar depth value, then the pixel and the neighboring pixel are grouped into a connected component.

19. The non-transitory computer readable medium as set forth in claim 18, further comprising detecting if a connected component is in motion between the depth image frame and a next depth image frame, the motion is detected by comparing a difference in the depth value of at least one pixel of the connected component in the depth image frame and a depth value of at least one pixel of the connected component in the next depth image frame, a connected component comprising user pixels is determined to be a user, a connected component comprising background pixels in motion is determined to be a user if a head of the user is detected, a connected component comprising unknown pixels is determined to be a user if a head of the user is detected.

20. The non-transitory computer readable medium as set forth in claim 19, wherein the region map comprises categorizing user pixels in the connected component in motion as foreground pixels if a user associated with the connected component in motion is enabled, the user pixels are categorized as background pixels if the user associated with the connected component is disabled.

21. The non-transitory computer readable medium as set forth in claim 17, further comprising a foreground region filling if a resolution of the depth image is less than a resolution of the color image, the foreground region filling comprising categorizing at least one unknown pixel as a foreground pixel if the unknown pixel is surrounded by foreground pixels.

22. The non-transitory computer readable medium as set forth in claim 17, wherein the unclear region band is created by expanding, on a user body, a contour line separating the foreground region and the background region on the user body outwards and inwards and expanding the unclear region band surrounding a head of the user.

23. The non-transitory computer readable medium as set forth in claim 17, wherein the distributing of the categorized unclear pixels of the unclear region band comprises performing a graph cut on unclear pixels in the unclear region and adjacent foreground and background pixels, the graph cut segments the unclear pixels into at least two subsets comprising a foreground region comprising foreground pixels and a background region comprising background pixels.