Geometry tracking

• US 9,792,479 B2
• Filed: 01/29/2014
• Issued: 10/17/2017
• Est. Priority Date: 11/29/2011
• Status: Active Grant

• Current Assignee: Lucasfilm Limited (The Walt Disney Company)
• Original Assignee: Lucasfilm Limited (The Walt Disney Company)
• Inventors: Mallet, Ronald, Bhat, Kiran S., Wooley, Kevin
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Azima, Shaghayegh
• Application Number:
  US14/167,838
• Publication Number:
  US 20140147014A1
• Time in Patent Office: 1,357 Days
• Field of Search: 382107, 382103, 382154, 345419, 345427
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models

Abstract

A method of motion capture may include accessing a 3D model of a subject, and associating the 3D model of the subject with a 2D representation of the subject in a plurality of frames. The method may also include identifying a change to the 2D representation of the subject between two or more of the plurality of frames, and deforming the 3D model in a virtual 3D space. In some embodiments, the deforming may be based on the identified change to the 2D representation and at least one constraint restricting how the 3D model can be deformed.

Claims

1. A computer implemented method of motion capture, the method comprising:

accessing a 3D model;

associating the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space;

identifying one or more motion vectors derived from a change to the position of the 2D representation of the subject between two or more of the plurality of frames; and

deforming the 3D model in a virtual 3D space, wherein the deforming is based on;

the identified change to the position of the 2D representation; and

at least one constraint restricting how the 3D model can be deformed,

wherein the at least one constraint comprises the one or more motion vectors.

2. The computer implemented method of motion capture of claim 1, wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

3. The computer implemented method of motion capture of claim 1, further comprising projecting the identified change onto the virtual 3D space as 3D constraints.

4. The computer implemented method of motion capture of claim 1, wherein deforming the 3D model in the virtual 3D space is performed in a single use of a non-linear solver.

5. The computer implemented method of motion capture of claim 1, identifying a change to the 2D representation of the subject is based on texture tracking.

6. The computer implemented method of motion capture of claim 1, wherein deforming the 3D model in the virtual 3D space comprises solving for control values for an animation rig that defines non-linear relationships between at least some of the control values.

7. The computer implemented method of motion capture of claim 6, wherein the animation rig comprises one or more selections from the group consisting of:

a facial rig, a body rig, and a blend shape.

8. The computer implemented method of motion capture of claim 1, wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.

9. The computer implemented method of motion capture of claim 1, further comprising:

accessing one or more reference frames in the plurality of frames, each reference frame showing the subject; and

tracking 2D changes in a second frame relative to a first frame includes using a reference frame as a key frame specifying texture locations in a future frame during the motion capture.

10. The computer implemented method of motion capture of claim 1, wherein identifying a change to the 2D representation of the subject between two or more of the plurality of frames comprises:

identifying a location of an occlusion object that indicates an occlusion of textures in a frame; and

ignoring textures in the frame that overlap a projection of the occlusion object in the frame.

11. The computer implemented method of motion capture of claim 1, further comprising:

reading a plurality of first camera parameters for a first frame in the plurality of frames, the first camera parameters corresponding to a physical state of a real camera when the real camera recorded the first frame; and

determining a plurality of second camera parameters for a second frame in the plurality of frames, the second camera parameters corresponding to a physical state of the real camera when the real camera recorded the second frame.

12. The computer implemented method of motion capture of claim 1, further comprising:

determining a freeform deformation to be applied to the 3D model; and

adding the freeform deformation to the 3D model.

13. A motion capture system, the system comprising:

a geometry tracker configured to;

access a 3D model;

associate the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space; and

deform the 3D model in a virtual 3D space, wherein the deforming is based on;

an identified change to the position of the 2D representation; and

at least one constraint restricting how the 3D model can be deformed, wherein the at least one constraint comprises one or more motion vectors; and

a texture tracker configured to;

identifying the one or more motion vectors derived from the change to the position of the 2D representation of the subject between two or more of the plurality of frames.

14. The motion capture system of claim 13, wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

15. The motion capture system of claim 13, wherein the geometry tracker is further configured to project the identified change onto the virtual 3D space as 3D constraints.

16. The motion capture system of claim 13, wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.

17. A non-transitory computer readable medium comprising instructions stored thereon which, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

accessing a 3D model;

associating the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space;

identifying one or more motion vectors derived from a change to the position of the 2D representation of the subject between two or more of the plurality of frames; and

deforming the 3D model in a virtual 3D space, wherein the deforming is based on;

the identified change to the position of the 2D representation; and

at least one constraint restricting how the 3D model can be deformed,

wherein the at least one constraint comprises the one or more motion vectors.

18. The non-transitory computer readable medium of claim 17, wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

19. The non-transitory computer readable medium of claim 17, wherein the instructions cause the one or more processing devices to perform operations further comprising projecting the identified change onto the virtual 3D space as 3D constraints.

20. The non-transitory computer readable medium of claim 17, wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.