Two-dimensional infrared depth sensing

• US 9,690,984 B2
• Filed: 04/14/2015
• Issued: 06/27/2017
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wei, Yichen, Izadi, Shahram, Tankovich, Vladimir, Butler, Ben, Keskin, Cem, Fanello, Sean Ryan Francesco, Barsoum, Emad, Stachniak, Simon P.
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US14/686,528
• Publication Number:
  US 20160307032A1
• Time in Patent Office: 805 Days
• Field of Search: 382154
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A signal encoding an infrared (IR) image including a plurality of IR pixels is received from an IR camera. Each IR pixel specifies one or more IR parameters of that IR pixel. IR-skin pixels that image a human hand are identified in the IR image. For each IR-skin pixel, a depth of a human hand portion imaged by that IR-skin pixel is estimated based on the IR parameters of that IR-skin pixel. A skeletal hand model including a plurality of hand joints is derived. Each hand joint is defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

Claims

1. An image processing method, comprising:

receiving, from an infrared (IR) camera, a signal encoding an IR image including a plurality of IR pixels, each IR pixel specifying IR intensity of that IR pixel;

identifying, in the IR image, IR-skin pixels that image a human hand;

for each IR-skin pixel, estimating a depth of a human hand portion imaged by that IR-skin pixel based on the IR intensity of that IR-skin pixel; and

deriving a skeletal hand model including a plurality of hand joints, each hand joint defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

2. The image processing method of claim 1, wherein deriving the skeletal hand model includes:

assembling a depth map including a depth for each IR-skin pixel; and

identifying, based on one or both of the IR-skin pixels and the depth map, one or more anatomical features of the human hand.

3. The image processing method of claim 2, wherein deriving the skeletal hand model includes estimating, based on one or both of the anatomical features and the depth map, a position of a joint of the human hand.

4. The image processing method of claim 3, wherein estimating the hand joint position includes:

computing a hash for the depth map;

selecting a previously-validated depth map from a database of previously-validated depth maps whose hash best corresponds to the computed hash of the depth map, each previously-validated depth map having a skeletal hand model including a plurality of hand joints; and

assigning a position of a hand joint from the hand model corresponding to the selected previously-validated depth map to the hand joint position.

5. The image processing method of claim 3, wherein deriving the skeletal hand model includes using at least one randomly perturbed hand joint position estimated from a previous-frame depth map to minimize an error function defined as a function of a spatial correspondence between the skeletal hand model and the IR-skin pixels.

6. The image processing method of claim 3, wherein deriving the skeletal hand model includes:

applying a previously-trained machine learning algorithm to one or both of the depth map and the estimated hand joint position; and

assigning at least one hand joint position determined by the previously-trained machine learning algorithm to the skeletal hand model, the previously-trained machine learning algorithm trained with training data including a training set of two-dimensional IR images and ground truth annotations of the training set of two-dimensional IR images.

7. The image processing method of claim 6, wherein the previously-trained machine learning algorithm receives as input at least one hand joint position estimated from a previous-frame depth map.

8. The image processing method of claim 1, wherein, for each IR-skin pixel, the depth of the human hand portion imaged by that IR-skin pixel is estimated via a previously-trained machine learning algorithm trained with two-dimensional (2D) IR images and three-dimensional (3D) depth maps annotated with ground truths.

9. The image processing method of claim 8, wherein the 2D IR images include synthetic IR images generated to simulate reflection of IR light by human skin, and/or

the 3D depth maps include synthetic depth maps generated to enable association of depths specified by the synthetic depth maps with corresponding IR intensities specified by the synthetic IR images.

10. The image processing method of claim 1, wherein deriving the skeletal hand model includes:

assembling a depth map including a depth for each IR-skin pixel using a dynamically selected machine learning algorithm previously trained with training data specific to a recognized pose of the human hand.

11. The image processing method of claim 1, wherein the IR-skin pixels are identified based on a predetermined correspondence between a range of IR intensities and human skin.

12. The image processing method of claim 1, wherein the IR-skin pixels are identified based on geometric analysis of the IR image.

13. A computing device, comprising:

an infrared (IR) camera interface configured to receive, from an IR camera, an IR image including a plurality of IR pixels, each IR pixel specifying an IR intensity of that IR pixel;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

identify, in the IR image, IR-skin pixels that image a human hand;

for each IR-skin pixel, estimate a depth of a human hand portion imaged by that IR-skin pixel based on the IR intensity of that IR-skin pixel; and

derive a skeletal hand model including a plurality of hand joints, each hand joint defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

14. The computing device of claim 13, wherein the instructions to derive the skeletal hand model include instructions to:

assemble a depth map including a depth for each IR-skin pixel; and

identify, based on one or both of the IR-skin pixels and the depth map, one or more anatomical features of the human hand.

15. The computing device of claim 14, wherein the instructions to assemble the depth map include instructions to use a dynamically selected machine learning algorithm previously trained with training data specific to a recognized pose of the human hand.

16. The computing device of claim 14, wherein the instructions to derive the skeletal hand model include instructions to estimate, based on one or both of the anatomical features and the depth map, a position of a joint of the human hand.

17. The computing device of claim 13, wherein the instructions to derive the skeletal hand model include instructions to use at least one hand joint position estimated from a previous-frame depth map.

18. The computing device of claim 13, wherein the instructions to identify the IR-skin pixels include instructions to identify the IR-skin pixels based on a predetermined correspondence between a range of IR intensities and human skin.

19. An image processing method, comprising:

receiving, from a camera, a signal encoding an image including a plurality of pixels, each pixel specifying an IR intensity of that pixel;

identifying, in the image, skin pixels that image a human hand;

for each skin pixel, estimating a depth of a human hand portion imaged by that skin pixel based on the IR intensity of that skin pixel; and

deriving a skeletal hand model including a plurality of hand joints, each hand joint defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

20. The method of claim 19, wherein the camera is selected from the group consisting of an infrared (IR) camera encoding an IR image including a plurality of IR pixels each specifying an IR intensity, an ultraviolet (UV) camera encoding a UV image including a plurality of UV pixels specifying one or more UV parameters, and a visible light (VL) camera encoding a VL image including a plurality of VL pixels specifying one or more VL parameters.