Depth Image Compression

• US 20130106994A1
• Filed: 11/01/2011
• Published: 05/02/2013
• Est. Priority Date: 11/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sharp, Toby, Shotton, Jamie Daniel Joseph
• Granted Patent: US 9,557,836 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Depth image compression is described for example, to enable body-part centers of players of a game to be detected in real time from depth images or for other applications such as augmented reality, and human-computer interaction. In an embodiment, depth images which have associated body-part probabilities, are compressed using probability mass which is related to the depth of an image element and a probability of a body part for the image element. In various examples, compression of the depth images using probability mass enables body part center detection, by clustering output elements, to be speeded up. In some examples, the scale of the compression is selected according to a depth of a foreground region and in some cases different scales are used for different image regions. In some examples, certainties of the body-part centers are calculated using probability masses of clustered image elements.

Claims

1. A computer-implemented method of compressing a depth image comprising probability information, the method comprising:

receiving an image comprising a plurality of image elements each having a depth value and a probability distribution over a plurality of classes representing the likelihood that the image element is a member of each of the classes;

for each image element, calculating a probability mass for each class, the probability mass being related to the depth of the image element and the probability of the class for the image element;

calculating, for each class, a plurality of output elements at lower resolution than the received image at least by aggregating the image elements on the basis of the probability mass.

2. A method as claimed in claim 1 each output element comprising any of:

an aggregate probability mass and a plurality of characteristics of a probability distribution;

a vector having a center of mass of a probability distribution in each of three dimensions and an aggregate probability mass;

a downsampled body part vector having a center of mass of a body part probability distribution in each of at least three dimensions and an aggregate probability mass.

3. A method as claimed in claim 1 the classes being any of:

body parts, object classes, foreground and background regions.

4. A method as claimed in claim 1 the aggregation comprising calculating a sum of a plurality of image elements each image element weighted by its probability mass;

and calculating an aggregate probability mass by aggregating the probability masses of the plurality of image elements.

5. A method as claimed in claim 1 wherein calculating the output elements is carried out in parallel for each class.

6. A method as claimed in claim 1 comprising selecting a scale according to depth of a foreground region of the depth image and calculating the output elements according to that scale.

7. A method as claimed in claim 1 comprising selecting a plurality of scales and calculating a first group of the output elements according to a first one of the plurality of scales and calculating a second group of the output elements according to a second one of the plurality of scales.

8. A method as claimed in claim 1 comprising clustering selected output elements at least partly on the basis of probability mass to obtain body part center positions where the classes are body parts.

9. A method as claimed in claim 8 using a mean shift process whereby each output element is repeatedly shifted according to a weighted aggregation of neighboring output elements.

10. A method as claimed in claim 9 comprising calculating a certainty for each body part center position by using probability masses of output elements which were shifted to a cluster for that body part in the mean shift process or by counting output elements in a cluster.

11. An apparatus for compressing a depth image comprising:

an input arranged to receive an image comprising a plurality of image elements each having a depth value and a probability distribution over a plurality of classes representing the likelihood that the image element is a member of each of the classes;

a processor arranged to calculate, for each image element, a probability mass for each class, the probability mass being related to the depth of the image element and the probability of the class for the image element;

the processor being arranged to calculate, for each class, a plurality of output elements at lower resolution than the received image by aggregating the image elements on the basis of the probability mass.

12. An apparatus as claimed in claim 11 the processor being arranged to calculate each output element such that it comprises any of:

an aggregate probability mass and a plurality of characteristics of a probability distribution;

a vector having a center of mass of a probability distribution in each of three dimensions and an aggregate probability mass;

a downsampled body part vector having a center of mass of a body part probability distribution in each of at least three dimensions and an aggregate probability mass.

13. An apparatus as claimed in claim 11 the processor being arranged to carry out the aggregation by calculating a sum of a plurality of image elements each image element weighted by its probability mass;

and to calculate an aggregate probability mass by aggregating the probability masses of the plurality of image elements.

14. An apparatus as claimed in claim 11 the processor being arranged to calculate the output elements in parallel for each class.

15. An apparatus as claimed in claim 11 the processor being arranged to select a scale according to a depth of a foreground region of the depth image and to calculate the output elements according to that scale.

16. An apparatus as claimed in claim 11 the processor being arranged to select a plurality of scales and to calculate a first group of the output elements according to a first one of the plurality of scales and to calculate a second group of the output elements according to a second one of the plurality of scales.

17. A computer-implemented method comprising:

receiving an image comprising a plurality of image elements each having a depth value and a probability distribution over a plurality of body parts representing the likelihood that the image element is a member of each of the body parts;

for each image element, calculating a probability mass for each body part, the probability mass being related to the depth of the image element and the probability of the body part for the image element;

calculating, for each body part, a plurality of output elements at a lower resolution than the received image by aggregating the image elements on the basis of the probability mass;

clustering output elements to find positions of centers of the body parts.

18. A method as claimed in claim 17 comprising determining a certainty for each of the positions of centers of the body parts by using probability masses of the output elements or by counting output elements in clusters.

19. A method as claimed in claim 17 comprising filtering output elements on the basis of probability mass before carrying out the clustering.

20. A method as claimed in claim 17 comprising selecting a scale according to depth of a foreground region of the depth image and calculating the output elements according to that scale.