Method and system for automatically calibrating HMD device

• US 10,365,481 B2
• Filed: 11/14/2016
• Issued: 07/30/2019
• Est. Priority Date: 07/27/2016
• Status: Active Grant

• Current Assignee: Brillio LLC
• Original Assignee: Brillio LLC
• Inventors: Thomas, Renji Kuruvilla, Jain, Gaurav, Vinmani, Karthik Gopalakrishnan, Gupta, Puneet
• Primary Examiner(s): Yodichkas, Aneeta
• Application Number:
  US15/351,238
• Publication Number:
  US 20180031829A1
• Time in Patent Office: 988 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2380/10 Automotive applications

Abstract

The proposed invention provides a method for calibrating a HMD device of a user in a vehicle. The method includes detecting user movements while viewing VR content in the vehicle and checking if these motions meet user movement criteria. The method also includes detecting if the vehicle is stationary or moving. Further, on detecting that the vehicle is stationary dynamically calibrating the HMD device based on the movements of the user and on detecting that the vehicle is in motion dynamically calibrating the HMD device based on vehicle movements and the movements of the user.

Claims

1. A method for calibrating a Head Mounted Display (HMD) device of a user in a vehicle, the method comprising:

detecting at least one of movements of a user and movements of the vehicle while viewing content in an immersive mode;

determining whether a first calibration criteria is met and at least one second calibration criteria is met, wherein the first calibration criteria is different than the at least one second calibration criteria; and

performing one of;

determining at least one of a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the HMD device and dynamically calibrating the HMD device by adjusting the content based on the at least one of the position of the user and the direction of the vehicle, in response to detecting that both the first calibration criteria and the at least one second calibration criteria are met, and

dynamically calibrating the HMD device by displaying the content in a non-immersive mode in response to detecting one of both the first calibration criteria and the at least one second calibration criteria are not met, and the first calibration criteria is met but the at least one second calibration criteria is not met.

2. The method of claim 1, wherein the first calibration criteria comprises at least one of a calibration counter, a calibration timer, and the at least one second calibration criteria comprises at least one of a rapidness of movements of the vehicle, a rapidness of the movements of the user, a type of content being displayed on the HMD device, and user preferences.

3. The method of claim 1, wherein determining at least one of the direction of the vehicle based on the movements of the vehicle and the position of the user based on the movements of the HMD device comprising:

determining at least one of sensor data of the vehicle based on the movements of the vehicle, and sensor data of the HMD device based on the movements of the user; and

determining at least one of the direction of the vehicle based on the sensor data of the vehicle and the position of the user based on the sensor data of the HMD device.

4. The method of claim 3, wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device and the sensor data of the vehicle comprises information of yaw, pitch and roll of at least one sensor of the vehicle.

5. A method for calibrating a Head Mounted Display (HMD) device of a user in a vehicle, the method comprising:

detecting movements of a user while viewing content in the vehicle;

determining that the movements of the user meets a user movement criteria;

detecting that the vehicle is one of stationary and moving; and

performing one of;

determining a position of the user based on the movement of the user and dynamically calibrating the HMD device by adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary, and

determining movements of the vehicle, determining a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the user and dynamically calibrating the HMD device by adjusting the content based on the direction of the vehicle and the position of the user, in response to detecting that the vehicle is moving.

6. The method of claim 5, wherein the HMD device is calibrated to seamlessly compensate for a change in position of content displayed in a field of view of the HMD device, wherein the change in the position of the content is formed by at least one of the movements of the user and the movements of the HMD device.

7. The method of claim 5, wherein calibrating the HMD device by adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary comprising:

determining sensor data of the HMD device based on the movements of the user in response to detecting that the vehicle is stationary;

detecting whether a plurality of calibration criteria are met; and

performing one of;

calibrating the HMD device to position of the user, in response to determining that the plurality of calibration criteria are met, and

calibrating the HMD device to display the content in a non-immersive mode in response to determining that the plurality of calibration criteria are not met.

8. The method of claim 7, wherein the plurality of calibration criteria comprises at least two of a rapidness of movements of the user, a calibration counter, a calibration timer, a type of content being displayed on the HMD device, and user preference.

9. The method of claim 7, wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device.

10. The method of claim 5, wherein dynamically calibrating the HMD device by adjusting the content based on the direction of the vehicle and the position of the user in response to detecting that the vehicle is moving comprising:

receiving sensor data of the vehicle based on the movements of the vehicle in response to detecting that the vehicle is moving;

determining that the movements of the vehicle meets a vehicle movement criteria;

determining sensor data of the HMD device based on the movements of the user;

determining a correlation between the sensor data of the HMD device and the sensor data of the vehicle based on the direction of the vehicle and the position of the user;

determining whether a plurality of calibration criteria are met; and

performing one of;

calibrating the HMD device based on the correlation in response to determining that the plurality of calibration criteria are met, and

calibrating the HMD device to display the content in a non-immersive mode in response to determining that the plurality of calibration criteria are not met.

11. The method of claim 10, wherein the plurality of calibration criteria comprises at least two of a rapidness of movements of the vehicle, a rapidness of movements of the user, a calibration counter, a calibration timer and a type of content being displayed on the HMD device and user preference.

12. The method of claim 10, wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device and the sensor data of the vehicle comprises information of yaw, pitch and roll of at least one sensor of the vehicle.

13. An apparatus for automatic calibration, comprising

a memory;

a processor, coupled to the memory; and

a calibration unit, configured to

detect at least one of movements of a user and movements of the vehicle while viewing content in an immersive mode;

determine whether a first calibration criteria is met and at least one second calibration criteria is met, wherein the first calibration criteria is different than the at least one second calibration criteria; and

performing one of;

determine at least one of a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the HMD device and dynamically calibrate the HMD device by adjusting the content based on the at least one of the position of the user and the direction of the vehicle, in response to detecting that both the first calibration criteria and the at least one second calibration criteria are met, and

dynamically calibrate the HMD device by displaying the content in a non-immersive mode in response to detecting one of both the first calibration criteria and the at least one second calibration criteria are not met, and the first calibration criteria is met but the at least one second calibration criteria is not met.

14. The apparatus of claim 13, wherein the first calibration criteria comprises at least one of a calibration counter, a calibration timer, and the at least one second calibration criteria comprises at least one of a rapidness of movements of the vehicle, a rapidness of the movements of the user, a type of content being displayed on the HMD device, and user preferences.

15. The apparatus of claim 13, wherein determine at least one of the direction of the vehicle based on the movements of the vehicle and the position of the user based on the movements of the HMD device comprising:

determine at least one of sensor data of the vehicle based on the movements of the vehicle, and sensor data of the HMD device based on the movements of the user; and

determine at least one of the direction of the vehicle based on the sensor data of the vehicle and the position of the user based on the sensor data of the HMD device.

16. The apparatus of claim 15, wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device and the sensor data of the vehicle comprises information of yaw, pitch and roll of at least one sensor of the vehicle.

17. A Head Mounted Display (HMD) device for automatic calibration, comprising:

a calibration log;

a processor; and

a calibration unit, in communication with the processor and the calibration log, configured to;

detect movements of a user while viewing content in the vehicle,

determine that the movements of the user meets a user movement criteria,

detect that the vehicle is one of stationary and moving, perform one of;

determine a position of the user based on the movement of the user and dynamically calibrate the HMD device by adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary, and

determine movements of the vehicle, determining a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the user and dynamically calibrate the HMD device by adjusting the content based on the direction of the vehicle and the position of the user, in response to detecting that the vehicle is moving.

18. The HMD device of claim 17, wherein the HMD device is calibrated to seamlessly compensate for a change in position of content displayed in a field of view of the HMD device, wherein the change in the position of the content is caused by at least one of the movements of the user and the movements of the HMD device.

19. The HMD device of claim 17, wherein calibrating the HMD device by adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary comprising:

determining sensor data of the HMD device based on the movements of the user in response to detecting that the vehicle is stationary;

detecting whether a plurality of calibration criteria are met; and

performing one of;

calibrating the HMD device to position of the user, in response to determining that the calibration criteria is met, and

calibrating the HMD device to display the content in a non-immersive mode in response to determining that the calibration criteria is not met.

20. The HMD device of claim 19, wherein the calibration criteria comprises at least two of a rapidness of movements of the user, a calibration counter, a calibration timer, a type of content being displayed on the HMD device, and a user preference.

21. The HMD device of claim 19, wherein the sensor data of the HMD device comprises information of a yaw, a pitch and a roll of at least one sensor of the HMD device.

22. The HMD device of claim 17, wherein the calibration unit configured to dynamically calibrate the HMD device by adjusting the content based on the direction of the vehicle and the position of the user in response to detecting that the vehicle is moving comprising:

receiving sensor data of the vehicle based on the movements of the vehicle in response to detecting that the vehicle is moving;

determining that the movements of the vehicle meets a vehicle movement criteria;

determining sensor data of the HMD device based on the movements of the user;

determining a correlation between the sensor data of the HMD device and the sensor data of the vehicle based on the direction of the vehicle and the position of the user;

determining whether a calibration criteria is met; and

performing one of;

calibrating the HMD device based on the correlation in response to determining that the calibration criteria is met, and

calibrating the HMD device to display the content in a non-immersive mode in response to determining that the calibration criteria is not met.

23. The HMD device of claim 22, wherein the calibration criteria comprises a at least two of a rapidness of movements of the vehicle, a rapidness of movements of the user, a calibration counter, a calibration timer, a type of content being displayed on the HMD device and user preference.

24. The HMD device of claim 22, wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device and the sensor data of the vehicle comprises information of yaw, pitch and roll of at least one sensor of the vehicle.

25. A system for automatic calibration, comprising:

a vehicle comprising an in-vehicle sensor unit configured to sense movements of the vehicle; and

a HMD device displaying an immersive view of content, the HMD device is configured to;

detect movements of a user while viewing the content in the vehicle;

determine that the movements of the user meets a user movement criteria;

detect that the vehicle is one of stationary and moving; and

performing one of;

determine a position of the user based on the movement of the user and dynamically calibrating the HMD device by adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary, and

determine movements of the vehicle, determine a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the user, and dynamically calibrate the HMD device by adjusting the content based on the direction of the vehicle and the position of the user, in response to detecting that the vehicle is moving.

26. The system of claim 25, wherein adjusting the content based on the position of the user, in response to detecting that the vehicle is stationary comprising:

determine sensor data of the HMD device based on the movements of the user in response to detecting that the vehicle is stationary;

detect whether a plurality of calibration criteria are met, wherein a first calibration criteria from plurality of calibration criteria comprises at least one of a calibration counter, a calibration timer, and at least one second calibration criteria plurality of calibration criteria comprises at least one of a rapidness of movements of the vehicle, a rapidness of the movements of the user, a type of content being displayed on the HMD device, and user preferences; and

perform one of;

calibrate the HMD device to position of the user, in response to determining that the plurality of calibration criteria are met, and

calibrate the HMD device to display the content in a non-immersive mode in response to determining that the plurality of calibration criteria are not met.

27. The system of claim 25, wherein the HMD device is configured to dynamically calibrate the HMD device by adjusting the content based on the direction of the vehicle and the position of the user in response to detecting that the vehicle is moving comprising:

receive sensor data of the vehicle based on the movements of the vehicle in response to detecting that the vehicle is moving;

determine that the movements of the vehicle meets a vehicle movement criteria;

determine sensor data of the HMD device based on the movements of the user;

determine a correlation between the sensor data of the HMD device and the sensor data of the vehicle based on the direction of the vehicle and the position of the user;

determine whether a plurality of calibration criteria are met, wherein a first calibration criteria from plurality of calibration criteria comprises at least one of a calibration counter, a calibration timer, and at least one second calibration criteria plurality of calibration criteria comprises at least one of a rapidness of movements of the vehicle, a rapidness of the movements of the user, a type of content being displayed on the HMD device, and user preferences; and

perform one of;

calibrate the HMD device based on the correlation in response to determining that the plurality of calibration criteria are met, and

calibrate the HMD device to display the content in a non-immersive mode in response to determining that the plurality of calibration criteria are not met.

28. A system for automatic calibration, comprising:

a vehicle comprising an in-vehicle sensor unit configured to sense movements of the vehicle; and

a HMD device displaying an immersive view of content, the HMD device configured to;

detect at least one of movements of a user and movements of the vehicle while viewing the content in an immersive mode;

determine whether a whether a first calibration criteria is met and at least one second calibration criteria is met, wherein the first calibration criteria is different than the at least one second calibration criteria, wherein the first calibration criteria comprises at least one of a calibration counter, a calibration timer, and wherein the at least one second calibration criteria comprises at least one of a rapidness of movements of the vehicle, a rapidness of the movements of the user, a type of content being displayed on the HMD device, and user preferences, and wherein; and

performing one of;

determine at least one of a direction of the vehicle based on the movements of the vehicle and a position of the user based on the movements of the HMD device and dynamically calibrate the HMD device by adjusting the content based on the at least one of the position of the user and the direction of the vehicle, in response to detecting that both the first calibration criteria and the at least one second calibration criteria are met, and

dynamically calibrate the HMD device by displaying the content in a non-immersive mode in response to detecting one of both the first calibration criteria and the at least one second calibration criteria are not met, and the first calibration criteria is met but the at least one second calibration criteria is not met.

29. The system of claim 28, wherein determine at least one of the direction of the vehicle based on the movements of the vehicle and the position of the user based on the movements of the HMD device comprises:

determine at least one of sensor data of the vehicle based on the movements of the vehicle, and sensor data of the HMD device based on the movements of the user wherein the sensor data of the HMD device comprises information of yaw, pitch and roll of at least one sensor of the HMD device and the sensor data of the vehicle comprises information of yaw, pitch and roll of at least one sensor of the vehicle; and

determine at least one of the direction of the vehicle based on the sensor data of the vehicle and the position of the user based on the sensor data of the HMD device.