Apparatus and method for capturing the expression of a performer

• US 8,194,093 B2
• Filed: 09/15/2004
• Issued: 06/05/2012
• Est. Priority Date: 09/15/2004
• Status: Active Grant

• Current Assignee: Rearden Mova, LLC
• Original Assignee: OnLive Incorporated
• Inventors: Perlman, Stephen G., Pearce, Kenneth A., Cotter, Tim S., LaSalle, Greg, Speck, John
• Primary Examiner(s): Broome, Said
• Application Number:
  US10/942,413
• Publication Number:
  US 20060055699A1
• Time in Patent Office: 2,820 Days
• Field of Search: None
• US Class Current: 345/589
• CPC Class Codes: B33Y 50/00 Data acquisition or data processing for additive manufacturing
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 using acquisition arrangements

Abstract

A method is described comprising: applying a series of curves on specified regions of a performer'"'"'s face; tracking the movement of the series of curves during a motion capture session; and generating motion data representing the movement of the performer'"'"'s face using the tracked movement of the series of curves.

Claims

1. A method, comprising:

specifying a predefined facial curve pattern comprising a mesh of intersecting curves, wherein each individual curve in the mesh is uniquely identifiable;

applying the predefined facial curve pattern to a performer'"'"'s face;

storing the predefined facial curve pattern as curve data on a storage device, the curve data usable by a camera controller;

processing sequences of real-time visual images taken of said curves as said curves move during a motion capture session, including, processing visual images taken from regions of the curves residing between intersection points amongst the curves, said processing including correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves, and intersection points between the curves, the operation of processing further comprising;

(i) identifying a first set of positions of the curves, the first set of positions of the curves forming a first set of surface tiles having curved edges; and

(ii) identifying a second set of positions of the curves, the second set of positions of the curves forming a second set of surface tiles having curved edges;

generating motion data from the recorded first and second sets of positions of the curves captured in the sequences of real-time visual images, the motion data representing movement of the performer'"'"'s face during the motion capture session; and

storing the motion data on a storage device, the motion data usable to graphically reconstruct movement of the curves during the motion capture session;

wherein a first set of one or more cameras capture a first sequence of real-time video images of a adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance; and

wherein specifying the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein the predefined facial curve pattern is generated such that no curve can intersect another curve of the same curve type.

2. The method of claim 1 wherein said predefined facial curve pattern includes a ring substantially around a feature of said performer'"'"'s face.

3. The method of claim 2 wherein said pattern further comprises a ring substantially around an eye or mouth of said performer, said ring being intersected at various locations around said ring by one or more different curves.

4. The method of claim 1 wherein said mesh of intersecting curves reside on a nose of said performer.

5. The method as in claim 1 wherein the predefined facial curve pattern defines different colors for one or more of the curves, the different colors usable to distinguish and uniquely identify each of the curves, wherein the curve data indicates a particular color for each of the uniquely identifiable curves, and wherein correlating comprises matching the color of each of the curves captured in the sequences of real-time video images with the color indicated in the curve data.

6. The method as in claim 1 further comprising:

correlating each of the captured real-time video images from the first orientation and the second orientation with the curve data stored on one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves from the first orientation and the second orientation;

generating first motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images from the first orientation;

generating second motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images from the second orientation; and

storing the first motion data and the second motion data on a storage device, the first and second motion data usable for reconstructing the facial movements of the performer during the performance.

7. The method as in claim 6 further comprising:

graphically reconstructing movement of the curves during the motion capture session using a combination of the first motion data and the second motion data.

8. The method as in claim 1 wherein the length of each of the line segments is selected based on a desired resolution.

9. The method as in claim 1 wherein quantizing each of the curves into a plurality of line segments comprises:

identifying a series of points on each of the curves; and

combining the points to form vertices of a series of adjacent triangles.

10. The method of claim 6 further comprising:

generating third motion data having a three-dimensional (3D) representation of the motion of the curves and the intersection points between the curves across the sequences of real-time video images, the third motion data being generated using a combination of the first motion data and the second motion data; and

storing the third motion data on a storage device, the third motion data usable for reconstructing 3D facial movements of the performer during the performance.

11. A method for performing motion capture using curves comprising:

specifying a predefined facial curve pattern comprising a set of intersecting curves, wherein each individual curve in the set is uniquely identifiable;

adjusting the predefined facial curve pattern based on a topology of a performer'"'"'s face and applying the adjusted, predefined facial curve pattern to the performer'"'"'s face;

storing the adjusted, predefined facial curve pattern on a storage device, the curve data usable by a camera controller;

tracking the movement of the series of curves of the adjusted, predefined facial curve pattern on the performer'"'"'s face during a motion capture session by performing the operations of;

capturing sequences of real-time video images of the adjusted, predefined facial curve pattern as the performer makes facial movements during a performance; and

correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves;

generating motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images; and

storing the motion data on a storage device, the motion data usable for reconstructing the facial movements of the performer during the performance;

wherein a first set of one or more cameras capture a first sequence of real-time video images of the adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance;

wherein generating the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein the predefined curve pattern is generated such that no curve can intersect another curve of the same curve type.

12. The method as in claim 11 further comprising:

correlating each of the captured real-time video images from the first orientation and the second orientation with the curve data stored on one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves from the first orientation and the second orientation;

generating first motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images from the first orientation;

generating second motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images from the second orientation; and

storing the first motion data and the second motion data on a storage device, the first and second motion data usable for reconstructing the facial movements of the performer during the performance.

13. The method as in claim 12 further comprising:

graphically reconstructing movement of the curves during the motion capture session using a combination of the first motion data and the second motion data.

14. The method as in claim 11 wherein the length of each of the line segments is selected based on a desired resolution.

15. The method as in claim 11 wherein quantizing each of the curves into a plurality of line segments comprises:

identifying a series of points on each of the curves; and

combining the points to form vertices of a series of adjacent triangles.

16. The method of claim 12 further comprising:

generating third motion data having a three-dimensional (3D) representation of the motion of the curves and the intersection points between the curves across the sequences of real-time video images, the third motion data being generated using a combination of the first motion data and the second motion data; and

storing the third motion data on a storage device, the third motion data usable for reconstructing 3D facial movements of the performer during the performance.

17. A method, comprising:

specifying a predefined facial curve pattern comprising a mesh of intersecting curves, wherein each individual curve in the mesh is uniquely identifiable;

applying the predefined facial curve pattern to a performer'"'"'s face;

storing the predefined facial curve pattern as curve data on a storage device, the curve data usable by a camera controller;

processing sequences of real-time visual images taken of said curves as said curves move during a motion capture session, including, processing visual images taken from regions of the curves residing between intersection points amongst the curves, said processing including correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves, and intersection points between the curves, the operation of processing further comprising;

(i) identifying a first set of positions of the curves, the first set of positions of the curves forming a first set of surface tiles having curved edges; and

(ii) identifying a second set of positions of the curves, the second set of positions of the curves forming a second set of surface tiles having curved edges;

generating motion data from the recorded first and second sets of positions of the curves captured in the sequences of real-time visual images, the motion data representing movement of the performer'"'"'s face during the motion capture session; and

storing the motion data on a storage device, the motion data usable to graphically reconstruct movement of the curves during the motion capture session;

wherein a first set of one or more cameras capture a first sequence of real-time video images of a adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance; and

wherein specifying the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein the curve data comprises;

a list of curves organized by type, with each curve having a unique identifier associated therewith;

for each curve, an ordered list of other curves that it intersects along its length; and

a list of surface patches, each defined by the curves that make up its sides.

18. A method, comprising:

specifying a predefined facial curve pattern comprising a mesh of intersecting curves, wherein each individual curve in the mesh is uniquely identifiable;

applying the predefined facial curve pattern to a performer'"'"'s face;

storing the predefined facial curve pattern as curve data on a storage device, the curve data usable by a camera controller;

processing sequences of real-time visual images taken of said curves as said curves move during a motion capture session, including, processing visual images taken from regions of the curves residing between intersection points amongst the curves, said processing including correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves, and intersection points between the curves, the operation of processing further comprising;

(i) identifying a first set of positions of the curves, the first set of positions of the curves forming a first set of surface tiles having curved edges; and

(ii) identifying a second set of positions of the curves, the second set of positions of the curves forming a second set of surface tiles having curved edges;

generating motion data from the recorded first and second sets of positions of the curves captured in the sequences of real-time visual images, the motion data representing movement of the performer'"'"'s face during the motion capture session; and

storing the motion data on a storage device, the motion data usable to graphically reconstruct movement of the curves during the motion capture session;

wherein a first set of one or more cameras capture a first sequence of real-time video images of a adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance; and

wherein specifying the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein generating motion data further comprises;

quantizing each of the curves into a plurality of line segments between each of the intersection points; and

defining a plurality of triangles using the endpoints of each of the line segments and the intersection points of the curves as vertices of the triangles.

19. A method for performing motion capture using curves comprising:

specifying a predefined facial curve pattern comprising a set of intersecting curves, wherein each individual curve in the set is uniquely identifiable;

adjusting the predefined facial curve pattern based on a topology of a performer'"'"'s face and applying the adjusted, predefined facial curve pattern to the performer'"'"'s face;

storing the adjusted, predefined facial curve pattern on a storage device, the curve data usable by a camera controller;

tracking the movement of the series of curves of the adjusted, predefined facial curve pattern on the performer'"'"'s face during a motion capture session by performing the operations of;

capturing sequences of real-time video images of the adjusted, predefined facial curve pattern as the performer makes facial movements during a performance; and

correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves;

generating motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images; and

storing the motion data on a storage device, the motion data usable for reconstructing the facial movements of the performer during the performance;

wherein a first set of one or more cameras capture a first sequence of real-time video images of the adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance;

wherein generating the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein the predefined facial curve pattern defines different colors for one or more of the curves, the different colors usable to distinguish and uniquely identify each of the curves, wherein the curve data indicates a particular color for each of the uniquely identifiable curves, and wherein correlating comprises matching the color of each of the curves captured in the sequences of real-time video images with the color indicated in the curve data.

20. A method for performing motion capture using curves comprising:

specifying a predefined facial curve pattern comprising a set of intersecting curves, wherein each individual curve in the set is uniquely identifiable;

adjusting the predefined facial curve pattern based on a topology of a performer'"'"'s face and applying the adjusted, predefined facial curve pattern to the performer'"'"'s face;

storing the adjusted, predefined facial curve pattern on a storage device, the curve data usable by a camera controller;

tracking the movement of the series of curves of the adjusted, predefined facial curve pattern on the performer'"'"'s face during a motion capture session by performing the operations of;

capturing sequences of real-time video images of the adjusted, predefined facial curve pattern as the performer makes facial movements during a performance; and

correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves;

generating motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images; and

storing the motion data on a storage device, the motion data usable for reconstructing the facial movements of the performer during the performance;

wherein a first set of one or more cameras capture a first sequence of real-time video images of the adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance;

wherein generating the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

wherein the curve data comprises;

a list of curves organized by type, with each curve having a unique identifier associated therewith;

for each curve, an ordered list of other curves that it intersects along its length; and

a list of surface patches, each defined by the curves that make up its sides.

21. A method for performing motion capture using curves comprising:

specifying a predefined facial curve pattern comprising a set of intersecting curves, wherein each individual curve in the set is uniquely identifiable;

adjusting the predefined facial curve pattern based on a topology of a performer'"'"'s face and applying the adjusted, predefined facial curve pattern to the performer'"'"'s face;

storing the adjusted, predefined facial curve pattern on a storage device, the curve data usable by a camera controller;

tracking the movement of the series of curves of the adjusted, predefined facial curve pattern on the performer'"'"'s face during a motion capture session by performing the operations of;

capturing sequences of real-time video images of the adjusted, predefined facial curve pattern as the performer makes facial movements during a performance; and

correlating each of the captured real-time video images with the curve data read by one or more of the camera controllers to uniquely identify each of the curves and intersection points between the curves;

generating motion data representing movement of each of the curves and the intersection points between the curves across the sequences of real-time video images; and

storing the motion data on a storage device, the motion data usable for reconstructing the facial movements of the performer during the performance;

wherein a first set of one or more cameras capture a first sequence of real-time video images of the adjusted, predefined facial curve pattern from a first orientation as the performer makes facial movements during a performance and a second set of one or more cameras capture a second sequence of real-time video images of the adjusted, predefined facial curve pattern from a second orientation as the performer makes facial movements during a performance;

wherein generating the predefined facial curve pattern comprises defining a plurality of curve types and categorizing each of the uniquely identifiable curves as one of the defined curve types, the curve types including contours, radials, and transition curves;

quantizing each of the curves into a plurality of line segments between each of the intersection points; and

defining a plurality of triangles using the endpoints of each of the line segments and the intersection points of the curves as vertices of the triangles.