Head mounted system to collect facial expressions

• US 20190029528A1
• Filed: 09/29/2018
• Published: 01/31/2019
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Granted Patent: US 10,376,153 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 2562/0276 comprising a thermosensitive compound
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/748 Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/0265 Handheld, portable ear thermometers G01J5/049
• CPC Class Codes: G01J 5/12 using thermoelectric elements, e.g. thermocouples
• CPC Class Codes: G01J 5/485 Temperature profile

Abstract

A head mounted system (HMS) configured to collect facial expressions of the user wearing the HMS. The HMS includes a frame and at least four cameras coupled to the frame. First and second cameras capture the user'"'"'s right and left eyebrows, and third and fourth cameras capture the right and left sides of the user'"'"'s upper lip. An optional computer utilizes the images captured by the cameras to detect facial expressions, microexpressions, and/or to improve the user'"'"'s emotional awareness.

Claims

1. A head mounted system (HMS) configured to collect facial expressions of a user wearing the HMS, comprising:

a frame configured to be worn on the user'"'"'s head;

first and second cameras coupled to the frame, at locations to the right and to the left of the symmetry axis that divides the face to the right and left sides, respectively, which are less than 15 cm away from the user'"'"'s right and left pupils, respectively;

the first and second cameras are oriented such that at least portions of the user'"'"'s right and left eyebrows are in the fields of view (FOVs) of the first and second cameras, respectively, and the user'"'"'s left and right oral commissures are not in the FOVs of the first and second cameras, respectively; and

third and fourth cameras coupled to the frame, at locations to the right and to the left of the symmetry axis, respectively, and less than 15 cm away from the user'"'"'s upper lip;

the third and fourth cameras are oriented such that at least portions of the right and left sides of the user'"'"'s upper lip are in the FOVs of the third and fourth cameras, respectively, and the user'"'"'s left and right eyebrows are not in the FOVs of the third and fourth cameras, respectively.

2. The HMS of claim 1, wherein the facial expressions are microexpressions, and at least one of the cameras is configured to have at least a portion of the user'"'"'s right cheek in its FOV, and that portion of the user'"'"'s right cheek enables a microexpression analyzer to identify a raised right cheek.

3. The HMS of claim 2, wherein at least one of the cameras is configured to have at least a portion of the user'"'"'s left cheek in its FOV, and that portion of the user'"'"'s left cheek enables a microexpression analyzer to identify a raised left cheek.

4. The HMS of claim 1, wherein at least one of the cameras is configured to have at least a portion of the user'"'"'s chin cheek in its FOV, and that portion of the user'"'"'s chin enables a microexpression analyzer to identify a raised chin.

5. The HMS of claim 1, wherein the facial expressions are microexpressions, and further comprising a processor configured to utilize a machine learning trained classifier to identify a microexpression expressed by the user.

6. The HMS of claim 5, wherein the processor is further configured to extract vision-related features from data derived from images captured by the first and second cameras, the machine learning trained classifier is trained to identify a microexpression that relates to the upper part of the face from the vision-related features, and the vision-related features comprise at least one of temporal features and features derived from locations of facial landmarks identified in the first and second images.

7. The HMS of claim 5, wherein the processor is further configured to extract vision-related features from data derived from images captured by the third and fourth cameras, the machine learning trained classifier is trained to identify a microexpression that relates to the lower part of the face from the vision-related features, and the vision-related features comprise at least one of temporal features and features derived from locations of facial landmarks identified in the third and fourth images.

8. The HMS of claim 1, further comprising a fifth camera coupled to the frame at a location that is less than 10 cm away from the user'"'"'s right pupil;

the fifth camera is oriented such that the lower orbital part of the user'"'"'s orbicularis oculi muscle that surrounds the user'"'"'s right eye is in the FOV of the fifth camera, and the user'"'"'s left oral commissure is not in the FOV of the fifth camera.

9. The HMS of claim 1, wherein at least one of the first to fourth cameras is selected from at least one of the following types of cameras:

(i) a depth camera configured to take measurements indicative of distances of objects relative to the camera, (ii) an extended depth of field camera that can capture in focus objects that are 2 to 5 cm from the first camera, (iii) a light field camera, and (iv) a camera that utilizes at least one of the following techniques to achieve an extended depth of field;

wavefront coding, diffusion coding, coded aperture, multiple apertures, and lens array.

10. The HMS of claim 1, further comprising a structured light pattern projector;

wherein the first camera is configured to capture a distorted pattern of the reflected structured light, the structured light pattern projector transmits in wavelength longer than 700 nm, and further comprising a processor configured to calculate at least one of depth and movement from the captured distorted pattern in order to identify the facial expressions.

11. The HMS of claim 1, further comprising an eye tracker and a processor;

the eye tracker is configured to track gaze of the user in order to identify an object the user is looking at;

the processor is configured to decode a facial expression of the user based on data received from at least one of the first and second cameras, and to associate the decoded facial expression with the object.

12. The HMS of claim 1, further comprising a wearable wireless transceiver configured to connect the HMS with a computer that is not carried by the user;

and further comprising a facial expression compressor configured to receive the pictures from the first and second cameras, and to extract points of interest that represent movements of the eyebrows, wherein storing the points of interest requires less than 10% of the storage required to store the pictures from the first and second cameras, and transmitting the points of interest to the computer.

13. The HMS of claim 1, further comprising a display and a controller;

the display is coupled to the frame and configured to present digital content to the user;

wherein the controller is configures to command the first and second cameras to capture images at a higher rate when the display presents an object that is expected to cause the user to have a noticeable emotional response, compared to the rate of capturing images by the first and second cameras when the display presents an object that is not expected to cause the user to have a noticeable emotional response.

14. The HMS of claim 1, further a processor configured to receive data derived from images captured by the cameras, identify a facial expression expressed by the user, and provide to the user a feedback related to the identified facial expression.

15. The HMS of claim 1, further comprising brainwave electrodes coupled to the frame and a computer configured to calculate affective response of the user based on data received from the brainwave electrodes and the cameras.

16. The HMS of claim 1, further comprising a computer configured to identify brow contraction based on data captured by the first and second cameras;

the computer is further configured to alert the user to release the contraction after identifying a contraction above a first threshold that is held for duration longer than a second threshold.

17. The HMS of claim 1, further comprising a fifth camera coupled to the frame and located behind the user'"'"'s ears, and a computer configured to estimate the user'"'"'s posture and facial expression based on data captured by the first, second, third, fourth, and fifth cameras.

18. The HMS of claim 1, further comprising a computer configured to estimate facial expressions of the user based on a model of a human face parameterized by expression and data extracted from images captured by the cameras.

19. The HMS of claim 18, wherein the computer is further configured to render an avatar of the user based on the facial expressions.

20. The HMS of claim 18, wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user his/her own avatar;

whereby seeing his/her own avatar helps the user to understand how he/she looks to others.