CAPTURING SURFACE IN MOTION PICTURE

• US 20080100622A1
• Filed: 09/05/2007
• Published: 05/01/2008
• Est. Priority Date: 11/01/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Inventors: GORDON, Demian
• Granted Patent: US 8,330,823 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: H04N 23/40 Circuit details for pick-up tubes

Abstract

Capturing a surface in motion picture, including: covering a surface with a pattern formed of a marking material; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the pattern covering the surface; deriving a mesh object from the plurality of images for the each image frame; tracking the mesh object in each frame through the sequence of frames; and generating animation data modeling a characteristic of the surface using the tracked mesh object.

Claims

1. A method, comprising:

covering a surface with a pattern formed of a marking material;

acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the pattern covering the surface;

deriving a mesh object from the plurality of images for the each image frame;

tracking the mesh object in each frame through the sequence of frames; and

generating animation data modeling a characteristic of the surface using the tracked mesh object.

2. The method of claim 1, wherein the marking material conforms to the surface.

3. The method of claim 1, wherein the mesh object models the surface.

4. The method of claim 1, wherein the mesh object is defined by a plurality of vertices.

5. The method of claim 4, wherein said tracking the mesh object includes

tracking at least one vertex of said plurality of vertices.

6. The method of claim 1, wherein the animation data include labeled marker data in the form of at least one of:

skeleton data, FACS animation curves, and shape animation.

7. The method of claim 1, wherein the surface includes a surface of a person.

8. The method of claim 1, wherein the pattern includes a predetermined pattern.

9. The method of claim 1, wherein the pattern includes a random pattern.

10. A system, comprising:

an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a pattern disposed on a surface; and

a surface capture module configured to receive the sequence of image frames and generate animation data based on the pattern disposed on the surface.

11. The system of claim 10, wherein said surface capture module includes:

a mesh derivation module configured to receive the sequence of image frames and generate at least one sequence of mesh objects; and

a tracking module configured to receive the at least one sequence of mesh objects and generate animation data from tracking information.

12. The system of claim 11, wherein said at least one sequence of mesh objects includes

a mesh object generated for each image frame of the sequence of image frames.

13. The system of claim 12, wherein said mesh object generated for each image frame is defined by a plurality of vertices.

14. The system of claim 13, wherein said tracking module includes

a vertex tracking subunit configured to generate said tracking information by tracking at least one vertex of said plurality of vertices defining one mesh object through said at least one sequence of mesh objects.

15. The system of claim 13, wherein said tracking module includes

a facial action coding system (“

FACS”

) cleaning subunit configured to analyze spatial displacements of at least one vertex of said plurality of vertices according to predefined constraints.

16. The system of claim 12, wherein said tracking module includes

a topography subunit configured to generate the tracking information by recognizing at least one topographical feature of one mesh object and tracking the topographical feature through said at least one sequence of mesh objects.

17. The system of claim 12, wherein said tracking module includes

a texture tracking subunit configured to generate the tracking information by tracking a textural feature of one mesh object through said at least one sequence of mesh objects.

18. The system of claim 11, wherein said tracking module includes

a kinematic skeleton subunit configured to generate tracking rules based on skeletal model information.

19. A computer program, stored in a computer-readable storage medium, the program comprising executable instructions that cause a computer to:

acquire a sequence of image frames, each image frame of the sequence including a plurality of images of a pattern covering a surface;

derive a mesh object from the plurality of images for the each image frame;

track the mesh object in each frame through the sequence of frames; and

generate animation data modeling a characteristic of the surface using the tracked mesh object.

20. The computer program of claim 19, wherein the mesh object is defined by a plurality of vertices, and the executable instructions that cause a computer to track the mesh object includes executable instructions that cause a computer to track at least one vertex of the plurality of vertices.

21. The computer program of claim 19, wherein executable instructions that cause a computer to track the mesh object includes executable instructions that cause a computer to

recognize a topographical feature of the mesh object.

22. A method, comprising:

obtaining a 3-D model of an actor'"'"'s head;

unwrapping the 3-D model into a 2-D texture;

replacing the details of the actor'"'"'s facial features represented in the 2-D texture with a known pattern;

printing the 2-D texture with the known pattern onto a flexible material;

applying the flexible material to the actor'"'"'s face;

acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the actor'"'"'s face; and

deriving a mesh object representing the actor'"'"'s face from the sequence of image frames.

23. The method of claim 22, wherein the flexible material includes temporary tattoo paper.

24. The method of claim 22, wherein the flexible material includes a transparent material.

25. The method of claim 22, wherein obtaining a 3-D model of an actor'"'"'s head includes performing a laser scan of the actor'"'"'s head.