Depth image compression

• US 9,557,836 B2
• Filed: 11/01/2011
• Issued: 01/31/2017
• Est. Priority Date: 11/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sharp, Toby, Shotton, Jamie Daniel Joseph
• Primary Examiner(s): Holder, Anner
• Assistant Examiner(s): Adrovel, William
• Application Number:
  US13/286,966
• Publication Number:
  US 20130106994A1
• Time in Patent Office: 1,918 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Depth image compression is described for example, to enable body-part centers of players of a game to be detected in real time from depth images or for other applications such as augmented reality, and human-computer interaction. In an embodiment, depth images which have associated body-part probabilities, are compressed using probability mass which is related to the depth of an image element and a probability of a body part for the image element. In various examples, compression of the depth images using probability mass enables body part center detection, by clustering output elements, to be speeded up. In some examples, the scale of the compression is selected according to a depth of a foreground region and in some cases different scales are used for different image regions. In some examples, certainties of the body-part centers are calculated using probability masses of clustered image elements.

Claims

1. A computer-implemented method of compressing a depth image comprising probability information, the method comprising:

receiving an image comprising a plurality of image elements having depth values and probability distributions over a plurality of classes representing the likelihood that the image elements are members of the classes;

for the image elements, calculating probability masses for the classes, the probability masses calculated based on probability values of the image elements being related to the classes multiplied by squares of the depth values of the image elements;

calculating, a plurality of output elements at lower resolution than the received image at least through aggregating the image elements on the basis of the probability masses; and

clustering output elements to find positions of centers of the classes.

2. A method as claimed in claim 1, wherein the plurality of output elements comprises at least one of:

an aggregate probability mass and a plurality of characteristics of a probability distribution;

a vector having a center of mass of a probability distribution in three dimensions and an aggregate probability mass;

or a downsampled body part vector having a center of mass of a body part probability distribution in at least three dimensions and an aggregate probability mass.

3. A method as claimed in claim 1 the classes being at least one of:

body parts, object classes, foreground regions, or background regions.

4. A method as claimed in claim 1 the aggregation comprising:

calculating a sum of a plurality of image elements each image element weighted by its probability mass; and

calculating an aggregate probability mass by aggregating the probability masses of the plurality of image elements.

5. A method as claimed in claim 1 wherein calculating the output elements is carried out in parallel for the classes.

6. A method as claimed in claim 1 comprising selecting a scale according to depth of a foreground region of the depth image and calculating the output elements according to that scale.

7. A method as claimed in claim 1, further comprising:

selecting a plurality of scales and calculating a first group of the output elements according to a first one of the plurality of scales; and

calculating a second group of the output elements according to a second one of the plurality of scales.

8. A method as claimed in claim 1 wherein clustering comprises clustering selected output elements at least partly on the basis of the probability masses to obtain body part center positions where the classes are body parts.

9. A method as claimed in claim 8 using a mean shift process whereby the output elements are repeatedly shifted according to weighted aggregations of neighboring output elements.

10. A method as claimed in claim 9 comprising calculating a certainty for a body part center position using the probability mass of the plurality of output elements which were shifted to a cluster for that body part in the mean shift process or by counting output elements in a cluster.

11. An apparatus for compressing a depth image comprising:

an input arranged to receive an image comprising a plurality of image elements having depth values and probability distributions over a plurality of classes representing the likelihood that the image elements are members of the classes; and

a processor programmed to;

calculate probability masses for the classes, the probability masses being probability values multiplied by squares of the depth values of the image elements, the probability values being related to probabilities of the classes for the image elements;

calculate a plurality of output elements at lower resolution than the received image by aggregating the image elements on the basis of the probability masses; and

cluster output elements to find positions of centers of the classes.

12. An apparatus as claimed in claim 11 wherein calculation of the plurality of output elements comprises at least one of:

an aggregate probability mass and a plurality of characteristics of a probability distribution;

a vector having a center of mass of a probability distribution in three dimensions and an aggregate probability mass;

or a downsampled body part vector having a center of mass of a body part probability distribution in one of at least three dimensions and an aggregate probability mass.

13. An apparatus as claimed in claim 11 the processor being arranged to:

carry out the aggregation through calculating a sum of a plurality of image elements weighted by its probability mass; and

calculate an aggregate probability mass through aggregating the probability masses of the plurality of image elements in the image.

14. An apparatus as claimed in claim 11 the processor being arranged to determine a center of at least one of the image elements in the lower resolution based on at least some of the probability masses.

15. An apparatus as claimed in claim 11 the processor being arranged to select a scale according to a depth of a foreground region of the depth image and to calculate the output elements according to that scale.

16. An apparatus as claimed in claim 11 the processor being arranged to:

select a plurality of scales and to calculate a first group of the output elements according to a first one of the plurality of scales; and

calculate a second group of the output elements according to a second one of the plurality of scales.

17. A computer-implemented method comprising:

receiving an image comprising a plurality of image elements each having a depth value and a probability distribution over a plurality of body parts representing the likelihood that the image element is a member of each of the body parts;

for each image element, calculating a probability mass for each body part, the probability mass being a probability value multiplied by the square of the depth value of the image element, the probability value being related to a probability of the body part for the image element;

calculating, for each body part, a plurality of output elements at a lower resolution than the received image by aggregating the image elements on the basis of the probability mass;

clustering output elements to find positions of centers of the body parts.

18. A method as claimed in claim 17 comprising determining a certainty for each of the positions of centers of the body parts by using probability masses of the output elements or by counting output elements in clusters.

19. A method as claimed in claim 17 comprising filtering output elements on the basis of probability mass before carrying out the clustering.

20. A method as claimed in claim 17 comprising selecting a scale according to depth of a foreground region of the depth image and calculating the output elements according to that scale.