Virtual reality system for sensing facial movements

• US 6,121,953 A
• Filed: 02/06/1997
• Issued: 09/19/2000
• Est. Priority Date: 02/06/1997
• Status: Expired due to Fees

• Current Assignee: Modern Cartoons Limited
• Original Assignee: Modern Cartoons Limited
• Inventors: Walker, Chris
• Primary Examiner(s): Hjerpe, Richard A.
• Assistant Examiner(s): NGUYEN, FRANCIS N
• Application Number:
  US08/795,544
• Time in Patent Office: 1,321 Days
• Field of Search: 345/156, 345/7, 345/8, 345/425, 345/473, 345/330, 364/526, 364/566, 348/39, 348/36, 348/42, 359/350, 359/815, 600/485, 600/587, 559/630, 463/36, 382/276, 382/118, 382/308, 382/103, 2/421
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A system for manipulating a computer generated animation in real time that includes a head mount configured to inhibit relative movement between the head mount and head of a wearer is provided. The head mount is coupled to a computer that processes data generated by the head mount in real time. Data generated by the facial movements of a wearer of the head mount is processed in real time so that an object, such as a cartoon in a virtual reality program, has facial expressions analogous to the facial expressions of the wearer, as the wearer is performing. A plurality of cameras monitor different localized regions of the wearer'"'"'s face for sensing movements in these regions to generate data indicating movement. The cameras are attached to the head mount with flexible arms for precise positioning of the cameras. The cameras monitor each of the mouth, chin and cheek, and an eyebrow and forehead regions of the wearer'"'"'s face and one of the wearer'"'"'s eyes. A portion of reflective material adhered to the wearer'"'"'s face in each of the monitored regions enhances the camera'"'"'s ability to generate data indicative of the changing facial expressions of the wearer. While the facial expressions of the wearer are changing, data representative of the changing expressions is continuously processed, so the cartoon has facial expressions that continuously mimic the wearer'"'"'s.

Claims

1. A head mount for use in a virtual reality computer system, the head mount comprising:

a helmet for wearing the head mount on a head of a wearer, the helmet configured to engage the head for inhibiting relative movement therebetween and not occluding a forehead region of the wearer;

a plurality of facial movement detection means for monitoring movement in a plurality of facial regions, each of the detection means continuously sensing and generating movement data;

a plurality of positioning means for positioning each of the detecting means at one of the facial regions; and

means for continuously transmitting the movement data to the computer system.

2. The system of claim 1 wherein the helmet includes means for adjusting the circumference of the helmet for inhibiting movement between the head of the wearer and the head mount.

3. The system of claim 1 wherein the facial movement detecting means comprises a plurality of light sensitive image reception means, the image reception means monitoring changes in the amount of light received and indicating corresponding changing positions of the wearer'"'"'s epidermis.

4. The system of claim 1 further comprising at least one eye movement detecting means for continuously detecting eye movement and blinking of the wearer and for continuously generating movement data indicative of the eye movement and blinking.

5. The system of claim 1 wherein the means for transmitting further comprises a data cable for coupling the head mount to the computer system.

6. A head mount for use in a virtual reality computer system, the system including a computing means for continuously processing data generated by the head mount, the head mount comprising:

a helmet configured to inhibit relative movement between a head of a wearer and the head mount, the helmet including a brow portion configured to engage the frontal region and the forehead of the head to prevent movement therebetween and a means for adjusting a circumference of the helmet;

a plurality of facial movement detection means for monitoring movement in a plurality of facial regions, each of the detection means continuously sensing and generating movement data;

means for positioning each of the detection means; and

means for transmitting the movement data to the computing means, the movement data being continuously processed so that a computer generated object in the computing means has computer generated facial expressions analogous to natural facial expressions of the wearer.

7. The system of claim 6 wherein the detection means further comprises an eye movement detection means for continuously detecting eye movement and blinking and continuously generating movement data indicative of eye movement and blinking.

8. The system of claim 7 further comprising light source means for illuminating the facial regions, the light source means enhancing the detection means'"'"' ability to sense movement in the facial regions and to produce corresponding computer generated facial expressions in the computer generated object.

9. The system of claim 7 wherein the brow portion is substantially rigid and configured to form a compound curve for mating with a top portion of the forehead of the head to inhibit movement therebetween;

and

the circumference adjusting means further comprising an adjustable headband affixed to above the brow portion and configured to extend about the wearer'"'"'s head for adjusting the circumference of the helmet so that the head mount can accommodate different head sizes while being securely retained on each wearer'"'"'s head.

10. The system of claim 8 wherein detection means are provided for monitoring at least the mouth, cheek, and forehead regions, and at least one eye.

11. The system of claim 10 wherein the positioning means comprise a plurality of positionable arms coupled to the helmet, the positionable arms provided to position each of the facial movement detection means at the desired facial region and for maintaining position of the detection means.

12. The system of claim 11 further comprising a data cable for coupling the facial movement detection means to the computing means to provide a data path therebetween, the detection means continuously transmitting movement data to the computing means via the data cable.

13. A head mount for use in a virtual reality computer system, the system including a computing means for continuously processing data generated by the head mount, the head mount comprising:

a helmet configured to inhibit relative movement between the head of a wearer and the head mount, the helmet portion including a rigid brow portion curved in a base plane and in a transverse plane extending substantially perpendicularly to the base plane, such that the brow portion forms a compound curve configured to mate with the frontal region and forehead of the head of the wearer to engage the wearer'"'"'s head to inhibit movement therebetween, the helmet further including an adjustable headband attached to the brow portion and configured to extend about the occipital region of the wearer'"'"'s head, the headband being adjustable for adjusting the circumference of the helmet so that the head mount may be dimensioned to accommodate different head sizes while securely retaining the head mount on each wearer'"'"'s head;

a plurality of epidermis position indicating means adhered to the face of the wearer in a plurality of localized facial regions, movement of the indicating means indicating changing facial expressions;

a plurality of facial movement monitoring means for monitoring facial movements in the plurality of facial regions, each monitoring means continuously tracking movement of the adjacent position indicating means for continuously generating facial expression movement data representative of the changing facial expressions of the wearer;

an eye movement monitoring means for continuously monitoring eye movement and blinking in one eye of the wearer for continuously generating movement data indicative of eye movement and blinking;

light source means for illuminating the monitored regions of the wearer'"'"'s face and for illuminating the wearer'"'"'s monitored eye, the light source means enhancing the monitoring means ability to detect movement in the monitored facial regions and eye for generating representative facial expression and eye movement data;

a plurality of positionable flexible arms for precisely positioning each of the monitoring means above the desired facial region and eye to be monitored and for maintaining the monitoring means in the desired position; and

means for transmitting the movement data generated by each of the monitoring means to the computing means, wherein facial expression and eye movement data generated by the monitoring means is continuously transmitted to the computing means for tracking the changing facial expressions and eye movements of the wearer for continuously processing the data so that an object in a virtual reality program running on the computing means has facial expressions and eye movements analogous to the facial expressions and eye movements of the wearer as they are occurring on the wearer'"'"'s face.

14. The system of claim 13 further comprising:

each of the monitoring means comprising a camera for sensing light reflected into a lens thereof, each of the cameras monitoring the amount of light reflected into the lens thereof by the adjacent epidermis position indicating means to determine changes in the position of the indicating means for continuously generating facial expression data, the eye monitoring camera generating data indicative of eye movement and blinking due to the lack of light reflected into the lens thereof by the pupil of the eye; and

each of the epidermis position indicating means comprising a portion of reflective material adhered to the face of the wearer, portions of reflective material adhered to the wearer in a plurality of localized facial regions selected from the group consisting of the mouth, chin, cheek, an eyebrow, and forehead regions, such that facial expression data is generated by facial movements in each of the plurality of monitored facial regions and eye movement data is generated by the eye monitoring camera for continuously generating data representative of the changing facial expressions and eye movements of the wearer so that the object has facial expressions analogous to the facial expressions of the wearer as they are occurring.

15. The system of claim 14 further comprising:

the cameras comprising infrared sensitive cameras which filter out excessive ambient light, and

the light source means comprising a plurality of light emitting diodes affixed adjacent to the lens of each camera and positioned thereabout, the light emitting diodes providing a focused light source for illuminating the adjacent facial region to be monitored.

16. The system of claim 15 wherein a plurality of light emitting diodes are affixed adjacent to the lens of the eye monitoring camera and positioned thereabout for inundating the eye with light, the number of light emitting diodes ranging from approximately four to approximately eight diodes.

17. A head mount for use in a virtual reality computer system, the system including a computing means for continuously processing data generated by the head mount, the head mount comprising:

a helmet configured to inhibit relative movement between the head of a wearer and the head mount, the helmet portion including a rigid brow portion curved in a base plane and in a transverse plane extending substantially perpendicularly to the base plane, such that the brow portion forms a compound curve configured to mate with the frontal region and forehead of the head of the wearer to engage the wearer'"'"'s head to inhibit movement therebetween, the helmet further including an adjustable headband attached to the brow portion and configured to extend about the occipital region of the wearer'"'"'s head, the headband being adjustable for adjusting the circumference of the helmet so that the head mount may be dimensioned to accommodate different head sizes while securely retaining the head mount on each wearer'"'"'s head;

a plurality of epidermis position indicating means adhered to the face of the wearer in a plurality of localized facial regions, movement of the indicating means indicating changing facial expressions;

a plurality of facial movement monitoring means for monitoring facial movements in the plurality of facial regions, each monitoring means continuously tracking movement of the adjacent position indicating means for continuously generating facial expression movement data representative of the changing facial expressions of the wearer;

an eye movement monitoring means for continuously monitoring eye movement and blinking in one eye of the wearer for continuously generating movement data indicative of eye movement and blinking;

light source means for illuminating the monitored regions of the wearer'"'"'s face and for illuminating the wearer'"'"'s monitored eye, the light source means enhancing the monitoring means ability to detect movement in the monitored facial regions and eye for generating representative facial expression and eye movement data;

a plurality of positionable flexible arms for precisely positioning each of the monitoring means above the desired facial region and eye to be monitored and for maintaining the monitoring means in the desired position;

means for transmitting the movement data generated by each of the monitoring means to the computing means, wherein facial expression and eye movement data generated by the monitoring means is continuously transmitted to the computing means for tracking the changing facial expressions and eye movements of the wearer for continuously processing the data so that an object in a virtual reality program running on the computing means has facial expressions and eye movements analogous to the facial expressions and eye movements of the wearer as they are occurring on the wearer'"'"'s face; and

a data transmission frame secured to the helmet portion, the data frame comprising a hollow tube extending about the helmet portion having ends terminating adjacent to the brow portion, the ends of the frame configured to couple to ends of the repositionable arms for coupling the cameras to the computing means via the data cable, the repositionable arms being hollow for providing an enclosed data line path from an output of each camera through the hollow arm supporting the camera, through the hollow data frame, into the data cable, and to the computing means so that extraneous data lines do not interfere with the line of sight of the wearer or with movements made by the wearer.

18. The system of claim 17 wherein the repositionable flexible arms each comprise a repositionable hollow tube comprising a series of articulated ball and socket type couplings for positioning each of the cameras above the desired region to be monitored and for maintaining the desired position.

19. The system of claim 18 wherein the brow portion comprises a rigid polymeric material and the data frame comprises a lightweight aluminum alloy so that the head mount ranges in weight from approximately 0.5 to 1.5 pounds.

20. A method for providing data to a virtual reality computer system, comprising the steps of:

creating a computer data base representative of a virtual image;

obtaining a plurality of movement data sets from a plurality of movement detection means, each movement data set representative of movement of one of a plurality of facial regions on a user'"'"'s face;

interacting at least two of the movement data sets with the data base; and

creating an animated image from the interaction of the movement data sets and the data base and producing a corresponding facial expression on said animated image.

21. An apparatus for providing data to a virtual reality computer system, comprising:

a plurality of movement data sets, each movement data set representative of movement of one of a plurality of facial regions on a user'"'"'s face;

a transmission means for providing movement data from a cheek, an eye, a brow, a mouth, and chin; and

a computing means for receiving the movement data sets and creating an animated image from the interaction of the movement data sets and a computer data base representative of a virtual image.