MITIGATION OF HEAD-MOUNTED-DISPLAY IMPACT VIA BIOMETRIC SENSORS AND LANGUAGE PROCESSING

• US 20180256115A1
• Filed: 03/07/2017
• Published: 09/13/2018
• Est. Priority Date: 03/07/2017
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Inventors: Campbell, Kevin, Cvetkovic, Milan, Garza, Patricio, Johnson, Abinadi, Corenbaum, John, Lau, Quan
• Granted Patent: US 10,568,573 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 3/14 Arrangements specially adapted for eye photography apparatus or arrangements for taking photographs per se G03B
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/4023 Evaluating sense of balance
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/7275 Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor
• CPC Class Codes: A61B 5/7405 using sound
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 50/30 for calculating health indices; for individual health risk assessment

Abstract

Users of head-mounted displays (HMDs) often experience nausea and other symptoms often referred to a “virtual reality sickness.” Various health and biometric sensors are employed to gather baseline sensor measurements. The HMD or a device associated with it can then generate a range or threshold such that any future sensor measurements that fall outside of the range or cross the threshold suggest a health concern for the user of the HMD. The HMD can also be coupled to a microphone, and natural language processing (NLP) is employed to detect negative words or noises made by the user that could also suggest a health concern for the user. After detecting the health concern, the HMD can warn its user, can shut off certain functions, can communicate with an external device, or can trigger an alarm.

Claims

1. A method for illness mitigation during use of a head-mounted display (HMD), the method comprising:

receiving a baseline sensor measurement value from a health sensor, the baseline sensor measurement value measuring a health characteristic of a user of the head-mounted display (HMD);

generating a health threshold value based on the baseline sensor measurement;

receiving an active sensor measurement value from the health sensor, the active sensor measurement value measuring the health characteristic of the user of the head-mounted display (HMD) following generation of the health threshold value;

identifying that the active sensor measurement value has reached or crossed the health threshold value;

identifying a health concern associated with the user of the head-mounted display (HMD) based on the active sensor measurement value reaching or crossing the health threshold value; and

displaying a visual indicator on a display portion of the head-mounted display (HMD) in response to the identification of the health concern.

3. The method of claim 1, further comprising activating an audio output device associated with the head-mounted display (HMD) to play an audible warning in response to the identification of the health concern.

4. The method of claim 1, further comprising turning off one or more functions of the head-mounted display (HMD) in response to the identification of the health concern.

5. The method of claim 1, further comprising sending a transmission to an external device in response to the identification of the health concern, wherein the transmission includes at least one of a text-based notification, an audio-based notification, or a visual notification.

7. The method of claim 1, wherein the health sensor is a thermometer, and wherein the active sensor measurement value is a body temperature measurement value of the user of the head-mounted display (HMD).

8. The method of claim 1, wherein the health sensor is at least one of a moisture sensor or a galvanic skin response sensor, and wherein the active sensor measurement value is a skin sweat measurement value of the user of the head-mounted display (HMD).

9. The method of claim 1, wherein the health sensor is a camera, and wherein the active sensor measurement value concerns at least one eye of the user of the head-mounted display (HMD), and wherein the active sensor measurement value is at least one of a pupil dilation measurement value, an eye movement measurement value, or an eye moisture measurement.

10. The method of claim 1, wherein the health sensor is at least one of a accelerometer or a gyroscope, and wherein the active sensor measurement value concerns at least one of an orientation or a position of the user of the head-mounted display (HMD).

11. The method of claim 1, wherein the health sensor is a microphone, and wherein the active sensor measurement value corresponds to an attribute of a noise made by the user of the head-mounted display (HMD).

12. The method of claim 1, wherein identifying that the active sensor measurement value has reached or crossed the health threshold value includes identifying that the active sensor measurement value has reached or exceeded the health threshold value.

13. The method of claim 1, wherein identifying that the active sensor measurement value has reached or crossed the health threshold value includes identifying that the active sensor measurement value has reached or fallen below the health threshold value.

2. (canceled)

6. (canceled)

14. A head-mounted display (HMD) system for illness mitigation, the head-mounted display (HMD) system comprising:

a body to be worn on a head of a user;

a display integrated with the body;

a memory to store instructions;

a processor coupled to the memory, wherein execution of the instructions by the processor causes the head-mounted display (HMD) system to;

receive a baseline sensor measurement value from a health sensor, the baseline sensor measurement value measuring a health characteristic of the user,

generate a health threshold value based on the baseline sensor measurement,

receive an active sensor measurement value from the health sensor, the active sensor measurement value measuring the health characteristic of the user following generation of the health threshold value,

identify that the active sensor measurement value has reached or crossed the health threshold value,

identify a health concern associated with the user based on the active sensor measurement value reaching or crossing the health threshold value, and

display a visual indicator on a display integrated with the body in response to the identification of the health concern.

16. The head-mounted display (HMD) system of claim 14, further comprising a wireless communication interface, wherein the head-mounted display (HMD) system receives the baseline sensor measurement value and the active sensor measurement value from the health sensor wirelessly via the wireless communication interface.

17. The head-mounted display (HMD) system of claim 14, further comprising an alarm system that includes at least one of an audio output device or a visual output device, wherein execution of the instructions by the processor causes the head-mounted display (HMD) system to activate the alarm system in response to the identification of the health concern.

18. The head-mounted display (HMD) system of claim 17, wherein the alarm system outputs:

an audible warning output via the audio output device associated with the head-mounted display (HMD) system, or

a visual warning output via the visual output device on the display of the head-mounted display (HMD) system.

19. The head-mounted display (HMD) system of claim 14, further comprising a further comprising a wireless communication interface, wherein execution of the instructions by the processor causes the head-mounted display (HMD) system to send a transmission to an external device via the wireless communication interface in response to the identification of the health concern, wherein the transmission includes at least one of a text-based notification, an audio-based notification, or a visual notification.

15. (canceled)

20. A non-transitory computer readable storage medium having embodied thereon a program, wherein the program is executable by a processor to perform a method of illness mitigation during use of a head-mounted display (HMD), the method comprising:

receiving a baseline sensor measurement value from a health sensor, the baseline sensor measurement value measuring a health characteristic of a user of the head-mounted display (HMD);

generating a health threshold value based on the baseline sensor measurement;

receiving an active sensor measurement value from the health sensor, the active sensor measurement value measuring the health characteristic of the user of the head-mounted display (HMD) following generation of the health threshold value;

identifying that the active sensor measurement value has reached or crossed the health threshold value;

identifying a health concern associated with the user of the head-mounted display (HMD) based on the active sensor measurement value reaching or crossing the health threshold value; and

displaying a visual indicator on a display portion of the head-mounted display (HMD) in response to the identification of the health concern.