Capturing surface in motion picture

• US 8,330,823 B2
• Filed: 09/05/2007
• Issued: 12/11/2012
• Est. Priority Date: 11/01/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Inventors: Gordon, Demian
• Primary Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US11/850,546
• Publication Number:
  US 20080100622A1
• Time in Patent Office: 1,924 Days
• Field of Search: 348/162, 348/207.99, 348370-371
• US Class Current: 348/208.14
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: H04N 23/40 Circuit details for pick-up tubes

Abstract

Capturing a surface in motion picture, including: covering a surface with a pattern formed of a marking material; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the pattern covering the surface; deriving a mesh object from the plurality of images for the each image frame; tracking the mesh object in each frame through the sequence of frames; and generating animation data modeling a characteristic of the surface using the tracked mesh object.

Claims

1. A method, comprising:

covering a surface with a pattern formed of a marking material including material which operates in both dark and light lighting conditions to provide texture and lighting of the surface;

acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the pattern covering the surface,

wherein the plurality of images of the pattern is formed as a random pattern to capture motion and geometry of the surface;

deriving a mesh object from the plurality of images for the each image frame;

tracking a plurality of vertices of the mesh object in each image frame through the sequence of image frames; and

generating animation data modeling characteristics of the surface including the texture and lighting using the tracked plurality of vertices of the mesh object.

2. The method of claim 1, wherein the marking material conforms to the surface.

3. The method of claim 1, wherein the mesh object models the surface.

4. The method of claim 1, wherein the mesh object is defined by the plurality of vertices.

5. The method of claim 1, wherein the animation data include

labeled marker data in the form of at least one of:

skeleton data, FACS animation curves, and shape animation.

6. The method of claim 1, wherein the surface includes a surface of a person.

7. The method of claim 1, wherein the marking material includes at least one of quantum nanodot material and fluorescent material.

8. The method of claim 1, wherein the marking material includes at least one of infra-red ink, infra-red paint, ultra-violet ink, ultra-violet paint, and glow-in-the dark substance.

9. A system, comprising:

an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a pattern disposed on a surface with a marking material including material which operates in both dark and light lighting conditions to provide texture and lighting of the surface, and

wherein the plurality of synchronized images of the pattern is formed as a random pattern to capture motion and geometry of the surface; and

a surface capture module configured to receive the sequence of image frames and generate animation data based on the pattern disposed on the surface,

said surface capture module comprising;

a mesh derivation module configured to receive the sequence of image frames and generate at least one sequence of mesh objects; and

a tracking module configured to receive and track a plurality of vertices of the at least one sequence of mesh objects, and to generate the animation data modeling characteristics of the surface including the texture and lighting using the tracked plurality of vertices.

10. The system of claim 9, wherein said at least one sequence of mesh objects includes

a mesh object generated for each image frame of the sequence of image frames.

11. The system of claim 10, wherein said mesh object generated for each image frame is defined by a plurality of vertices.

12. The system of claim 11, wherein said tracking module includes

a facial action coding system (“

FACS”

) cleaning subunit configured to analyze spatial displacements of at least one vertex of said plurality of vertices according to predefined constraints.

13. The system of claim 10, wherein said tracking module includes

a topography subunit configured to generate tracking information by recognizing at least one topographical feature of one mesh object and tracking the at least one topographical feature through said at least one sequence of mesh objects.

14. The system of claim 10, wherein said tracking module includes

a texture tracking subunit configured to generate tracking information by tracking a textural feature of one mesh object through said at least one sequence of mesh objects.

15. The system of claim 9, wherein said tracking module includes

a kinematic skeleton subunit configured to generate tracking rules based on skeletal model information.

16. A non-transitory computer-readable storage medium for storing a computer program, the program comprising executable instructions that cause a computer to:

acquire a sequence of image frames, each image frame of the sequence including a plurality of images of a pattern covering a surface, and the pattern is formed of a marking material including material which operates in both dark and light lighting conditions to provide texture and lighting of the surface, and

wherein the plurality of images of the pattern is formed as a random pattern to capture motion and geometry of the surface;

derive a mesh object from the plurality of images for the each image frame;

track a plurality of vertices of the mesh object in each image frame through the sequence of image frames; and

generate animation data modeling characteristics of the surface including the texture and lighting using the tracked plurality of vertices of the mesh object.

17. The non-transitory computer-readable storage medium of claim 16, wherein the mesh object is defined by a plurality of vertices.

18. The non-transitory storage medium of claim 16, wherein the executable instructions that cause the computer to track the mesh object includes executable instructions that cause the computer to

recognize a topographical feature of the mesh object.

19. A method, comprising:

obtaining a 3-D model of an actor'"'"'s head;

unwrapping the 3-D model into a 2-D texture;

replacing the details of the actor'"'"'s facial features represented in the 2-D texture with a random pattern to capture motion and geometry of the facial features;

printing the 2-D texture with the random pattern onto a flexible material,

wherein the flexible material includes material which operates in both dark and light lighting conditions to provide texture and lighting of the surface;

applying the flexible material to the actor'"'"'s face;

acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the random pattern covering the actor'"'"'s face;

deriving a mesh object representing the actor'"'"'s face from the sequence of image frames; and

tracking vertices of the mesh object in each image frame of the sequence of image frames.

20. The method of claim 19, wherein the flexible material includes temporary tattoo paper.

21. The method of claim 19, wherein the flexible material includes a transparent material.

22. The method of claim 19, wherein obtaining a 3-D model of the actor'"'"'s head includes performing a laser scan of the actor'"'"'s head.