Subject segmentation and tracking using 3D sensing technology for video compression in multimedia applications

• US 7,203,356 B2
• Filed: 04/11/2003
• Issued: 04/10/2007
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Gokturk, Salih Burak, Rafii, Abbas
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/411,674
• Publication Number:
  US 20030235341A1
• Time in Patent Office: 1,460 Days
• Field of Search: 382/282, 382/154, 382/103, 382165-166, 382115-118, 382/168, 382/171, 382/173, 382/190, 382232-251, 348/14.01, 348/14.16, 348169-174, 348/211.12, 37524001-24025, 362/464, 702/72, 702/57, 356/3.03, 356 3- 301, 356/3.13, 345/419
• US Class Current: 382/154
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic

Abstract

Three-dimensional position information is used to segment objects in a scene viewed by a three dimensional camera. At one or more instances of an interval, the head location of the user is determined. Object-based compression schemes are applied on the segmented objects and the detected head.

Claims

1. A method for enabling segmentation of objects and subjects in an image captured by a camera system capable of obtaining both luminosity and depth information, the method comprising:

obtaining position information for a plurality of discrete regions on at least one of a head and a body part of a subject and on a background behind said subject;

segmenting the image into at least a first segment corresponding to a first object in the image and into a second segment corresponding to a second object in the image, at least one of the first object and the second object corresponding to at least one of the head and the body part of the subject;

identifying a position and a shape of at least one of the head and the body part of the subject; and

applying a first compression scheme to the first segment of the image, and a second compression scheme to the second segment of the image.

2. The method of claim 1, wherein the step of obtaining position information includes using one of a time-of-flight based three-dimensional sensor, a stereo-vision three-dimensional sensor, a structured light three-dimensional sensor, a depth-from-focus three-dimensional sensor.

3. The method of claim 1, wherein the step of segmenting the image includes identifying at least one of a foreground object and a background object from the image.

4. The method of claim 1, wherein the step of segmenting the image includes clustering pixels together based at least partially on a depth of a discrete region of each of the first object and the second object from a reference.

5. The method of claim 4, wherein clustering pixels includes executing a k-means algorithm.

6. The method of claim 4, wherein clustering pixels includes executing a split-and-merge algorithm.

7. The method of claim 4, wherein clustering pixels includes grouping a plurality of detected depth values as either a foreground object or a background object.

8. The method of claim 7, wherein grouping a plurality of detected depth values includes using a prior expectation of depth values of the foreground and background objects.

9. The method of claim 7, wherein grouping a plurality of detected depth values includes using k-means algorithm with a k value of 2.

10. The method of claim 1, wherein identifying a position and a shape of at least one of the head and the body part of the subject includes the steps of:

determining an initial location, a shape and a size of said head; and

tracking the head to determine the location of the head in a series of consecutive frames.

11. The method of claim 10, wherein determining an initial location, a shape and a size of said head includes the steps of:

obtaining a row-sum and a column-sum of a plurality of foreground pixels in the image;

identifying multiple edges of said row-sum and column-sum distributions;

finding one or more bounding boxes of the head by an intersection of said multiple edges of said row-sum and column-sum distributions.

12. The method of claim 10, wherein determining an initial location, a shape and a size of said head includes the steps of:

obtaining a plurality of edge values from said image, the plurality of edge values corresponding to multiple edges of said objects;

clustering the plurality of edge values that have similar depth values into one or more groups of edge values;

fitting an ellipse to each of the groups of edge values;

identifying which group in the one or more groups has an ellipse of a designated criteria; and

designating the group with the ellipse with the designated criteria as the head of the subject.

13. The method of claim 12, wherein obtaining a plurality of edge values from said image includes using a standard edge detection algorithm on said image.

14. The method of claim 12, wherein clustering the plurality of edge values includes using k-means algorithm with a predefined value of k.

15. The method of claim 12, wherein fitting an ellipse to each of the groups of edge values includes using linear least-square fit to one or more of the edge'"'"'s locations.

16. The method of claim 10, wherein determining an initial location, a shape and a size of said head includes executing an intensity based face detection algorithm.

17. The method of claim 10, wherein determining an initial location, a shape and a size of said head includes modeling a human head as one or more three-dimensional ellipsoids.

18. The method of claim 17, further comprising of the steps of:

clustering adjacent pixels forming image and which have similar depth values into groups; and

fitting a three-dimensional ellipsoid to each said group of said pixels.

19. The method of claim 10, wherein tracking the head includes the steps of:

determining a head search space using a current frame to be used for a next frame;

obtaining one or more edges of the image in said head search space;

fitting an ellipse to the one or more edges.

20. The method of claim 19, wherein determining a head search space includes defining said head search space as a band around the ellipse for the head location in said current frame.

21. The method of claim 19, wherein fitting an ellipse to the one or more edges includes executing a linear least-square fit algorithm using the one or more edges.

22. The method of claim 1, wherein the step of applying the first compression scheme and the second compression scheme includes using different compression algorithms, wherein the compression algorithm of the first compression scheme has a quality that is different than the compression algorithm of the second scheme.

23. The method of claim 1, wherein the step of applying the first compression scheme and the second compression scheme includes applying the first compression scheme to a foreground portion of the image, and a second compression scheme to a background portion of the image.

24. The method of claim 23, wherein the first compression scheme has a first compression algorithm, and wherein the second compression scheme has a second compression algorithm that is different than the first compression algorithm, and wherein the second compression algorithm has a different quality than the first compression algorithm.

25. The method of claim 1, wherein the step of applying the first compression scheme and the second compression scheme includes applying on each segment one or a combination of compression schemes selected from a group consisting of (i) a two-dimensional discrete cosine transformation;

(ii) vector quantization;

(iii) wavelet transformation;

(iv) principal component analysis;

(v) motion compensated coding;

(vi) three-dimensional discrete cosine transformation;

(vii) three-dimensional principal component analysis; and

(viii) three-dimensional wavelet transformation.

26. The method of claim 1, wherein the step of applying the first compression scheme and the second compression scheme includes applying on the first segment and the second segment a protocol selected from the group of (i) Joint Photographic Experts Group (JPEG);

(ii) Joint Photographic Experts Group 2000 (JPEG2000);

(iii) Moving Picture Expert Group 2 (MPEG2);

(iv) Moving Picture Expert Group 4 (MPEG

     4);

(v) Moving Picture Expert Group 7 (MPEG

     7);

(vi) Moving Picture Expert Group 21 (MPEG

     21);

(vii) Audio Video Interleave (AVI);

(viii) protocols recommended by International Telecommunications Union (H.261 and H.263).

27. The method of claim 23, wherein the step of applying the first compression scheme and the second compression scheme includes applying on each segment one or a combination of compression schemes selected from a group consisting of (i) a two-dimensional discrete cosine transformation;

(ii) vector quantization;

(iii) wavelet transformation;

(iv) principal component analysis;

(v) motion compensated coding;

(vi) three-dimensional discrete cosine transformation;

(vii) three-dimensional principal component analysis; and

(viii) three-dimensional wavelet transformation.

28. The method of claim 23, wherein the step of applying the first compression scheme and the second compression scheme includes applying on the first segment and the second segment a protocol selected from the group of (i) Joint Photographic Experts Group (JPEG);

(ii) Joint Photographic Experts Group 2000 (JPEG2000);

(iii) Moving Picture Expert Group 2 (MPEG2);

(iv) Moving Picture Expert Group 4 (MPEG

     4);

(v) Moving Picture Expert Group 7 (MPEG

     7);

(vi) Moving Picture Expert Group 21 (MPEG

     21);

(vii) Audio Video Interleave (AVI);

(viii) protocols recommended by International Telecommunications Union (H.261 and H.263).

29. The method of claim 1, wherein the step of applying the first compression scheme and the second compression scheme includes using object-specific compression.

30. The method of claim 1, further comprising transmitting the image over a communication channel using at least the first compression scheme and the second compression scheme.

31. The method of claim 30, wherein said communication channel includes a channel selected from a group consisting of a telephone line, a local area network, a wide area network, a digital subscriber line, and a wireless media.

32. The method of claim 1, further comprising the step of storing the compressed first segment and the compressed second segment in a computer-readable medium.

33. The method of claim 32, wherein said computer-readable medium is selected from a group consisting of a hard disk, a storage tape, random access memory, and read-only memory.

34. The method of claim 30, further comprising:

transmitting the compressed image over a communication channel to a receiver, where the receiver includes a processor for decompressing compressed images;

receiving and decompressing the compressed image;

applying a background processing;

applying image processing to improve the decompressed image; and

displaying the image.

35. The method of claim 34, wherein applying a background processing includes substituting as a background, one of a static image, a static image of the background where the image was taken, a static video, and any static color.

36. The method of claim 34, wherein applying image processing includes applying image blending and feathering to the image.

37. A system for enabling segmentation of objects and subjects using images obtained by a three-dimensional sensor and enabling object-based compression of segmented said objects and subjects, the system comprising:

a host device including a processor and a memory, the memory storing one or more modules that are executable by the processor;

a camera system capable of obtaining an image with luminosity and depth information;

a communication interface for transmitting the image;

wherein the module is executable by the processor in order to;

segment the image into at least a first segment corresponding to a first object in the image and into a second segment corresponding to a second object in the image, at least one of the first object and the second object corresponding to at least one of the head and the body part of the subject;

identify a position and a shape of at least one of the head and the body part of the subject; and

apply a first compression scheme to the first segment of the image, and a second compression scheme to the second segment of the image.

38. The system of claim 37, wherein the host device is a device selected from a group consisting of a personal computer, a laptop computer, a personal digital assistant, a cellular phone, a land-line phone, a pen tablet computer;

and a screen.

39. The system of claim 38, further comprising a sensor to obtain depth information with the image, wherein the sensor is selected from a group consisting of a time-of-flight based three-dimensional sensor, a stereo-vision three-dimensional sensor, a structured light three-dimensional sensor, and a depth-from-focus three-dimensional sensor.

40. The system of claim 37, wherein the communication interface is selected from a group consisting of a telephone line, a local area network, a wide area network, a digital subscriber line, and a wireless medium.