Estimating posture using head-mounted cameras

• US 10,165,949 B2
• Filed: 06/14/2016
• Issued: 01/01/2019
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Frank, Ari M, Thieberger, Gil
• Primary Examiner(s): Shin, Soo
• Application Number:
  US15/182,592
• Publication Number:
  US 20160360970A1
• Time in Patent Office: 931 Days
• Field of Search: None
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 2562/0276 comprising a thermosensitive compound
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/748 Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/0265 Handheld, portable ear thermometers G01J5/049
• CPC Class Codes: G01J 5/12 using thermoelectric elements, e.g. thermocouples
• CPC Class Codes: G01J 5/485 Temperature profile

Abstract

System and method for estimating posture of a user based on a model and images captured by at least two head-mounted cameras that are worn by the user and are oriented downward such that portions of the user'"'"'s torso are in the respective fields of view of the cameras when the user stands up straight.

Claims

1. A system comprising:

a frame configured to be worn on a user'"'"'s head;

first and second cameras, physically coupled to the frame at locations that are to the right and to the left of a symmetry axis that divides the user'"'"'s face to its right and left sides, respectively, and less than 15 cm away from the user'"'"'s head;

wherein the first and second cameras are oriented downward such that portions of the user'"'"'s torso are in respective fields of view of the first and second cameras when the user stands upright; and

a computer configured to estimate an angle between the user'"'"'s head and the user'"'"'s torso based on data received from the first and second cameras and orientation of the frame.

2. The system of claim 1, wherein the computer is further configured to estimate posture of the user based on a model and the data received from the first and second cameras, and to train the model based on data received from the first and second cameras;

wherein the training of the model is performed under the assumption that the locations of the first and second cameras, relative to the user'"'"'s head, do not change by more than 5°

even when the user'"'"'s head performs wide angular and lateral movements.

3. The system of claim 2, wherein the first and second cameras are coupled to the frame at locations that are less than 5 cm away from the user'"'"'s head, and further comprising an inertial measurement unit (IMU) coupled to the frame and configured to measure orientation of the frame;

wherein the computer is further configured to utilize the orientation for training the model.

4. The system of claim 2, further comprising an external camera, which is not physically coupled to the frame, having a direct line of sight to the front side of the user'"'"'s torso;

wherein the computer is further configured to utilize data from the external camera to train the model.

5. The system of claim 1, wherein the first camera points at least 20°

to the right of the symmetry axis, and the second camera points at least 20°

to the left of the symmetry axis; and

wherein the first and second cameras are coupled to the frame at locations that are less than 5 cm away from the user'"'"'s head.

6. The system of claim 1, further comprising a third camera, physically coupled to the frame at a location behind an ear of the user;

wherein the third camera is oriented downwards such that a portion of the back of the user'"'"'s torso is in field of view of the third camera when the user stands upright; and

wherein the computer is configured to estimate posture of the user based on data received from the first, second and third cameras.

7. The system of claim 6, wherein the third camera is mounted on an extender that helps to prevent concealment of the portion of the user'"'"'s torso by at least one of the user'"'"'s head and hair.

8. The system of claim 1, further comprising third and fourth cameras, physically coupled to the right and left sides of the frame, respectively, at locations that enable them to capture the user'"'"'s hands when stretched horizontally to the sides;

wherein the computer is configured to estimate posture of the user based on data received from the first, second, third and fourth cameras.

9. The system of claim 1, further comprising a third camera, physically coupled to the frame and oriented such that at least a portion of at least one of the user'"'"'s eyebrows is in the field of view of the third camera;

wherein the computer is further configured to estimate a facial expression of the user based on a model of a human face parameterized by expression and data extracted from images captured by the third camera.

10. The system of claim 9, wherein the computer is further configured to estimate posture of the user based on a model and data received from the first and second cameras, and to render an avatar of the user based on the posture and the facial expression.

11. The system of claim 10, wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user his/her own avatar;

whereby seeing his/her own avatar helps the user to understand how he/she looks to others.

12. The system of claim 10, wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user at least two of his/her own avatars who imitate the user by doing what the user does.

13. The system of claim 9, wherein the computer is further configured to estimate posture of the user based on a model and data received from the first and second cameras, the frame belongs to an augmented reality device, and further comprising a communication module configured to send a representation of the posture and the facial expression to a second computer configured to render an avatar of the user;

wherein the second computer is further configured to receive additional measurements of the user from a second augmented reality device that is worn by a second user located nearby the user; and

the second computer is further configured to render, for the second user, an avatar of the user based on;

the additional measurements, the posture, and the facial expression.

14. The system of claim 1, wherein the first and second cameras are light field cameras.

15. A method comprising:

receiving images from first and second head mounted cameras that are located to the right and to the left of a symmetry axis that divides a user'"'"'s face to its right and left sides, respectively;

wherein the first and second head mounted cameras physically coupled to a frame worn on the user'"'"'s head, are located less than 15 cm away from the user'"'"'s head, and are oriented downward such that portions of the user'"'"'s torso are in respective fields of view of the first and second head mounted cameras when the user stands upright; and

estimating an angle between the user'"'"'s head and the user'"'"'s torso based on the data received from the first and second cameras and orientation of the frame.

16. The method of claim 15, further comprising training a model based on the data received from the first and second head mounted cameras;

wherein the training of the model is performed under the assumption that the locations of the first and second head mounted cameras, relative to the user'"'"'s head, do not change by more than 5°

even when the user'"'"'s head performs wide angular and lateral movements.

17. The method of claim 15, further comprising a third head mounted camera oriented such that at least a portion of at least one of the user'"'"'s eyebrows is in the field of view of the third head mounted camera;

and further comprising

estimating a facial expression of the user based on a model of a human face parameterized by expression and data extracted from images captured by the third head mounted camera.

18. The method of claim 17, further comprising estimating posture of the user based on a model and data received from the first and second cameras, and rendering an avatar of the user based on the posture and the facial expression.

19. The method of claim 15, wherein the first and second head mounted cameras belong to an augmented reality device or a virtual reality device, and further comprising showing the user his/her own avatar from the side;

whereby seeing his/her own avatar helps the user to understand how he/she looks to others.