3D skeletonization using truncated epipolar lines

• US 10,460,512 B2
• Filed: 11/07/2017
• Issued: 10/29/2019
• Est. Priority Date: 11/07/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Qiuyu
• Primary Examiner(s): Faulk, Devona E
• Assistant Examiner(s): Beard, Charles L
• Application Number:
  US15/806,145
• Publication Number:
  US 20190139297A1
• Time in Patent Office: 721 Days
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Technologies are provided for generating three-dimensional (3D) skeletons of target objects using images of the target objects captured from different viewpoints. Images of an object (such as a person) can be captured from different camera angles. Feature keypoints of the object can be identified in the captured images. Keypoints that identify a same feature in separate images can be correlated using truncated epipolar lines. For example, depth information for a keypoint can be used to truncate an epipolar line that is created using the keypoint. The correlated feature keypoints can be used to create 3D feature coordinates for the associated features of the object. A 3D skeleton can be generated using the 3D feature coordinates. One or more 3D models can be mapped to the 3D skeleton and rendered. The rendered one or more 3D models can be displayed on one or more display devices.

Claims

1. A computing device, comprising:

a processing unit; and

a memory;

the computing device programmed, via computer-executable instructions, to perform operations for performing three-dimensional skeletonization, the operations comprising;

receiving multiple two-dimensional images of a target object, wherein the multiple two-dimensional images depict the target object from different perspectives;

identifying corresponding keypoints of the target object in the multiple two-dimensional images, wherein the identifying comprises projecting one or more truncated epipolar lines onto one or more of the multiple two-dimensional images using one or more keypoints in another one or more of the multiple two-dimensional images, wherein creating a truncated epipolar line comprises subtracting a truncation distance from a distance value associated with one of the keypoints, wherein the distance value is retrieved from a depth map associated with one of the multiple two-dimensional images;

mapping the corresponding keypoints to three-dimensional coordinates in a three-dimensional space; and

generating a three-dimensional skeleton of the target object using the three-dimensional coordinates.

2. The computing device of claim 1, wherein:

the target object is a human; and

identifying the corresponding keypoints in the multiple two-dimensional images comprises identifying a body part of the human being in two of the two-dimensional images that depict the body part from different perspectives.

3. The computing device of claim 1, wherein identifying the corresponding keypoints comprises:

projecting an epipolar line on a first of the two-dimensional images using a keypoint in a second of the two-dimensional images;

truncating the epipolar line using the depth map associated with the first of the two-dimensional images; and

identifying a corresponding keypoint in the first of the two-dimensional images using the truncated epipolar line.

4. The computing device of claim 1, wherein the operations further comprise:

mapping a three-dimensional model to at least part of the three-dimensional skeleton; and

rendering the three-dimensional model.

5. The computing device of claim 1, wherein the mapping the corresponding keypoints to three-dimensional coordinates comprises:

triangulating a three-dimensional coordinate in the three-dimensional space using two corresponding keypoints.

6. The computing device of claim 5, wherein the triangulating comprises determining a direct linear transformation using the two corresponding keypoints.

7. The computing device of claim 1, wherein identifying the corresponding keypoints comprises:

determining confidence scores for detected keypoints in the multiple two-dimensional images with respect to a feature of the target object;

selecting a first keypoint in one of the multiple two-dimensional images with a highest confidence score;

selecting a second keypoint in another of the multiple two-dimensional images with a second-highest confidence score; and

projecting a truncated epipolar line on the another of the multiple two-dimensional images using the first keypoint; and

determining that the truncated epipolar line intersects the second keypoint.

8. The computing device of claim 1, wherein:

the multiple two-dimensional images are captured by multiple cameras configured to view the target object from different perspectives.

9. The computing device of claim 1, wherein the operations further comprise:

mapping a three-dimensional point cloud to the three-dimensional skeleton.

10. The computing device of claim 1, wherein:

the target object comprises a human; and

the operations further comprise;

identifying a specified priority region of the human in the multiple two-dimensional images; and

applying more computational resources to identifying and correlating features in the specified priority region than are applied to identifying and correlating features outside the specified priority region.

11. A method, implemented by a computing device comprising a processing unit and a memory, the method comprising:

receiving images depicting a person from multiple viewpoints;

identifying features of the person in the images;

correlating the identified features using one or more truncated epipolar lines, wherein creating a truncated epipolar line comprises subtracting a truncation distance from a distance value associated with one of the features, wherein the distance value is retrieved from a depth map associated with one of the images;

constructing three-dimensional feature coordinates for the identified features using the correlated identified features; and

generating a three-dimensional skeleton of the person using the three-dimensional feature coordinates.

12. The method of claim 11, further comprising:

generating a point cloud based on depth information captured by one or more depth cameras;

mapping the point cloud to the three-dimensional skeleton of the person; and

moving one or more points of the point cloud closer to a surface of a volumetric shape associated with one of the three-dimensional feature coordinates.

13. The method of claim 11, wherein:

the received images depict multiple people; and

the method further comprises;

using the one or more truncated epipolar lines to distinguish between separate features of the multiple people; and

generating separate three-dimensional skeletons for the multiple people.

14. The method of claim 11, further comprising:

receiving the depth map associated with the one of the images from a depth camera.

15. The method of claim 11, further comprising:

mapping a three-dimensional mesh to the generated three-dimensional skeleton;

applying a texture model to at least part of the three-dimensional mesh; and

rendering a projection of a three-dimensional scene comprising the three-dimensional mesh with the texture model applied.

16. The method of claim 11, wherein the constructing three-dimensional feature coordinates for the identified features comprises triangulating positions of the three-dimensional feature coordinates using two-dimensional positions of the correlated features in the received images.

17. A system for generating a three-dimensional skeleton of a human, the system comprising:

multiple camera devices configured to capture images of a human from different viewpoints; and

a computing device comprising a processor and a storage medium storing executable instructions that, when executed by the processor, program the computing device to perform operations, the operations comprising;

receiving the captured images from the multiple camera devices;

detecting keypoints in the captured images associated with body parts of the human;

correlating the detected keypoints, wherein the correlating comprises projecting one or more truncated epipolar lines onto one or more of the captured images using one or more keypoints detected in another one or more of the captured images to determine that keypoints in separate images, of the captured images, are associated with a same body part depicted from different viewpoints, wherein creating a truncated epipolar line comprises subtracting a truncation distance from a distance value associated with one of the keypoints, wherein the distance value is retrieved from a depth map associated with one of the captured images;

triangulating three-dimensional coordinates for the body parts using the correlated keypoints associated with the body parts; and

generating a three-dimensional skeleton comprising the three-dimensional coordinates.

18. The system of claim 17, wherein:

the camera devices comprise depth sensors configured to capture depth maps associated with the captured images; and

the operations further comprise;

receiving the depth maps from the camera devices; and

generating the one or more truncated epipolar lines using one or more of the depth maps received from the camera devices.

19. The system of claim 18, wherein the generating the one or more truncated epipolar lines comprises:

identifying two keypoints in two images, of the captured images, associated with a same body party type;

projecting an epipolar line on a first of the two images using the keypoint in a second of the two images;

truncating the epipolar line using a depth map associated with the first of the two images; and

determining that the truncated epipolar line intersects the keypoint in the first of the two images.

20. The system of claim 17, wherein the operations further comprise:

positioning a three-dimensional avatar using the generated three-dimensional skeleton; and

rendering a two-dimensional projection of the positioned three-dimensional avatar.