Mobile face capture and image processing system and method

• US 20050083248A1
• Filed: 08/09/2004
• Published: 04/21/2005
• Est. Priority Date: 12/22/2000
• Status: Abandoned Application

• Current Assignee: The Board of Trustees of Michigan State University (Michigan State University), University of Central Florida (State University System of Florida)
• Original Assignee: The Board of Trustees of Michigan State University (Michigan State University), University of Central Florida (State University System of Florida)
• Inventors: Reddy, Chandan K., Rolland, Jannick P., Figueroa, Miguel Villaneuva, Stockman, George C., Biocca, Frank
• Application Number:
  US10/914,621
• Publication Number:
  US 20050083248A1
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: A41D 31/32 Retroreflective
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

Image processing procedures include receiving at least two side view images of a face of a user. In other aspects, side view images are warped and blended into an output image of a face of a user as if viewed from a virtual point of view. In further aspects, a virtual video is produced in real time of output images from a video feed of side view images.

Claims

1. An image processing method, comprising:

receiving at least two side view images of a face of a user;

warping and blending the side view images into an output image of the face of the user as if viewed from a virtual point of view; and

producing a virtual video in real time of output images from a video feed of side view images.

2. The method of claim 1, further comprising:

accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points; and

warping and blending the side view images by texture mapping the side view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of side view images.

3. The method of claim 2, further comprising instantiating a mesh model for an individual user by obtaining scaling and deformation transformations.

4. The method of claim 3, further comprising obtaining scaling and deformation transformations by choosing special points by hand on a digital frontal and profile photo.

5. The method of claim 3, further comprising obtaining scaling and deformation transformations by choosing special points from the two side images of a neutral facial expression of the user captured by imaging components of a head mounted display worn by the user, and enabling the user wearing the head mounted display to make adjustments that apply various scaling and deformation transformations while viewing a resulting output image rendered by the head mounted display.

6. The method of claim 2, further comprising:

receiving a selection of a virtual point of view from which to view a three-dimensional model of the face of the user; and

rendering the three-dimensional model from the virtual point of view based on the selection.

7. The method of claim 6, further comprising applying a lighting model that determines how the three-dimensional model appears to be illuminated.

8. The method of claim 2, further comprising dynamically fitting parameters of the mesh model by optimizing similarity between a three-dimensional model rendered from a virtual point of view corresponding to an actual point of view of a side view image while using the parameters and the side view image.

9. The method of claim 8, further comprising fitting the parameters via hill-climbing so that incremental dynamic updates can be made to the model parameters for sequentially observed side video frames.

10. The method of claim 2, further comprising training a morphable model for dynamic use on a population of users, including capturing side views and a frontal view of a diverse set of training speakers.

11. The method of claim 10, further comprising:

hand labeling mesh points for a diverse set of faces and multiframe video recording; and

performing principal components analysis to obtain a minimum spanning dimensionality.

12. The method of claim 2, further comprising texture mapping triangles of the mesh model to stored face data as needed to fill in un-imaged patches.

13. The method of claim 1, further comprising:

accessing transformation tables for the side view images, wherein the transformation tables define rules for interpolating regions of the side view images into side portions of the output image;

warping the side view images based on the transformation tables, thereby producing the side portions of the output image; and

blending the side portions of the output image, thereby producing the output image.

14. The method of claim 13, further comprising creating the transformation tables by projecting a grid pattern onto a human face at least as if from the virtual point of view and mapping polygons of left and right calibration face images to corresponding polygons of the grid pattern.

15. The method of claim 13, wherein warping the side view images includes reconstructing coordinates in side portions of the output image by accessing corresponding locations in the transformation tables and retrieving pixels in the side view images using interpolation.

16. The method of claim 1, wherein receiving at least two side view images includes receiving side view images captured via at least two imaging components of a head mounted display worn by the user, said imaging components attached to said head mounted display unit and thereby obtaining fixed positions and orientations relative to the face of the user and adapted to receive at least two side views of the face of the user;

17. The method of claim 1, further comprising linearly smoothing the output image in order to smooth intensity across a vertical midline of the face.

18. An apparatus, comprising:

a head mounted display unit worn by a first user, the display unit rendering to the first user an output image of a face of a second user virtually interacting with the first user in a collaborative, virtual environment, wherein the output image has been formed, based on offset view images of the face of the second user, such that the face of the second user appears as if viewed from a virtual point of view; and

an input port receiving at least one of the following;

(a) offset view images of the face of the second user;

(b) user-specific scaling and deformation transformations specific to the second user;

(c) position of the face of the second user in a common coordinate system of the collaborative, virtual environment;

(d) a three-dimensional model of the face of the second user;

(e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the second user; and

(f) an output image of the face of the second user.

19. The apparatus of claim 18, further comprising:

an array of at least two imaging components having fixed positions and orientations relative to a face of the first user and adapted to receive at least two offset views of the face of the first user; and

an output port transmitting at least one of the following;

(a) offset view images of the face of the first user;

(b) user-specific scaling and deformation transformations specific to the first user;

(c) position of the face of the first user in the common coordinate system of the collaborative, virtual environment within which the first user and the second user virtually interact;

(d) a three-dimensional model of the face of the first user;

(e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the first user; and

(f) an output image of the face of the first user.

20. The apparatus of claim 19, further comprising an image processing module accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the first user, and combining the offset view images of the face of the first user by texture mapping the offset view images of the face of the first user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the first user, thereby forming the three-dimensional model of the face of the first user.

21. The apparatus of claim 20, wherein said image processing module is further adapted to select a virtual point of view from which to view the three-dimensional model of the face of the first user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the first user from the virtual point of view, thereby forming the output image of the face of the first user.

22. The apparatus of claim 21, wherein said image processing module is adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face of the first user.

23. The apparatus of claim 21, wherein said image processing module is adapted to apply a lighting model that determines how the three-dimensional model of the face of the first user appears to be illuminated.

24. The apparatus of claim 18, further comprising an image processing module adapted to select a virtual point of view from which to view the three-dimensional model of the face of the second user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the second user from the virtual point of view, thereby forming the output image of the face of the second user.

25. The apparatus of claim 24, wherein said imaging module is further adapted to access a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the second user, and combining the offset view images of the face of the second user by texture mapping the offset view images of the face of the second user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the second user, thereby forming the three-dimensional model of the face of the second user.

26. The apparatus of claim 24, wherein said image processing module is further adapted to apply a lighting model that determines how the three-dimensional model appears to be illuminated.

27. The apparatus of claim 18, further comprising an image processing module adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face.

28. An apparatus, comprising:

an array of at least two imaging components having fixed positions and orientations relative to a face of a first user and adapted to receive at least two offset views of the face of the first user; and

an output port transmitting at least one of the following;

(a) offset view images of the face of the first user;

(b) user-specific scaling and deformation transformations specific to the first user;

(c) position of the face of the first user in a common coordinate system of a collaborative, virtual environment within which the first user and a second user virtually interact;

(d) a three-dimensional model of the face of the first user;

(e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the first user; and

(f) an output image of the face of the first user, wherein the output image of the face of the first user has been formed by combining offset view images of the face of the first user into an output image of the face of the first user as if viewed from a virtual point of view.

29. The apparatus of claim 28, further comprising an image processing module accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the first user, and combining the offset view images of the face of the first user by texture mapping the offset view images of the face of the first user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the first user, thereby forming the three-dimensional model of the face of the first user.

30. The apparatus of claim 29, wherein said image processing module is further adapted to select a virtual point of view from which to view the three-dimensional model of the face of the first user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the first user from the virtual point of view, thereby forming the output image of the face of the first user.

31. The apparatus of claim 30, wherein said image processing module is adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face of the first user.

32. The apparatus of claim 29, wherein said image processing module is adapted to apply a lighting model that determines how the three-dimensional model of the face of the first user appears to be illuminated.

33. Computer software, comprising:

first instructions receiving at least two offset view images of a contoured structure;

second instructions forming, from the offset view images, an output image of the contoured structure as if viewed from a virtual point of view.

34. The computer software of claim 33, wherein said second instructions are adapted to recognize feature points of the contoured structure in the offset view images, to access a three-dimensional closed mesh model of feature points similar to the recognized feature points, and to texture map the offset view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images, thereby forming a three-dimensional model of the contoured structure.

35. The computer software of claim 33, wherein said second set of instructions is further adapted to select a virtual point of view from which to view the three-dimensional model, and to render the three-dimensional model from the virtual point of view, thereby forming the output image.

36. Computer software, comprising:

a first set of instructions receiving at least two offset view images of a contoured structure;

a second set of instructions recognizing feature points of the contoured structure in the offset view images, accessing a three-dimensional closed mesh model of feature points similar to the recognized feature points, and texture mapping the offset view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images, thereby forming a three-dimensional model of the contoured structure.