Head mounted display presentation adjustment

• US 9,928,653 B2
• Filed: 04/13/2015
• Issued: 03/27/2018
• Est. Priority Date: 04/14/2014
• Status: Active Grant

• Current Assignee: iOnRoad Technologies Limited (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Atsmon, Dan
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US14/685,482
• Publication Number:
  US 20150294505A1
• Time in Patent Office: 1,079 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Embodiments are disclosed for adjusting a presentation on a head-mounted display (HMD). In one or more example embodiments, a method of dynamically orienting a presentation of a HMD includes gathering HMD sensor data via at least one HMD sensor that is installed on an HMD worn by a driver of the vehicle and gathering vehicle sensor data via at least one vehicle mounted sensor mounted to the vehicle. The example method further includes performing an analysis of the HMD sensor data and of the vehicle sensor data to identify a difference between the HMD sensor data and the vehicle sensor data, and calculating, based on the difference, an orientation of the HMD device in relation to the vehicle. The method further includes adjusting a presentation of data on a display of the HMD device based on the orientation.

Claims

1. A method of dynamically orienting a presentation of a head-mounted display (HMD), comprising:

gathering HMD sensor data via at least one HMD sensor installed on an HMD worn by a driver of a vehicle, the HMD sensor data comprising an HMD image captured by the at least one HMD sensor installed on the HMD worn by the driver;

gathering vehicle sensor data via at least one vehicle-mounted sensor mounted on the vehicle;

performing an analysis of each of said HMD sensor data and of said vehicle sensor data to identify a difference between said HMD sensor data and said vehicle sensor data by identifying a difference in an apparent position of an object located in an environment of the vehicle and included in each of the HMD image and a vehicle image captured by the at least one vehicle-mounted sensor;

calculating, based on said difference, an orientation of said HMD in relation to said vehicle; and

adjusting a presentation of data on a display of said HMD based on said orientation.

2. The method of claim 1, wherein the vehicle sensor data comprises the vehicle image, and wherein the at least one HMD sensor is installed on a location of the HMD that is positioned at a front or side region of a head of the driver when the HMD is worn by the driver.

3. The method of claim 2, wherein the HMD image includes a driver control area and a scenery seen through at least part of a windshield of the vehicle, wherein the vehicle image includes at least a portion of the scenery imaged from a perspective of the vehicle sensor, and wherein the object is located in the scenery.

4. The method of claim 2, wherein the HMD image includes a driver control area comprising a set of size-known markers arranged in a predetermined arrangement, and where performing the analysis of the HMD sensor data comprises identifying an apparent size of each member of the set of size-known markers in the HMD image.

5. The method of claim 4, wherein identifying an apparent position and size of each member of the set of size-known markers comprises detecting the apparent position and size of each member of the set of size-known markers using a Scale-invariant feature transform (SIFT) process.

6. The method of claim 2, further comprising combining each of the HMD image and the vehicle image to create a stereoscopic image, wherein the vehicle image and the HMD image depict two different views of same scenery.

7. The method of claim 1, wherein the at least one HMD sensor includes a motion detector installed on the HMD worn by the driver and the HMD sensor data comprises a current measured HMD device motion determined via at least one HMD device motion detector, wherein the vehicle sensor data comprises a current measured vehicle motion determined via at least one vehicle motion detector, and wherein performing the analysis of the HMD sensor data and the vehicle sensor data further comprises identifying a difference between the current HMD device motion and the current vehicle motion, the motion detector including one or more of an accelerometer, a gyroscope, and a compass.

8. The method of claim 7, further comprising capturing the HMD image, via at least one HMD image sensor installed on the HMD, of a driver control area and a scenery seen through part of a windshield of the vehicle, capturing the vehicle image, via at least one vehicle imaging device mounted on the vehicle, of at least a portion of the scenery, wherein performing the analysis of the HMD sensor data and the vehicle sensor data further comprises identifying a difference between an apparent position of an object located in the scenery and imaged in both the HMD image and the vehicle image, and wherein calculating the orientation is further based on the apparent position.

9. The method of claim 7, further comprising capturing the HMD image, via at least one HMD image sensor installed on the HMD, of a driver control area comprising a set of size-known markers arranged in a predetermined arrangement, wherein performing the analysis of the HMD sensor data further comprises identifying an apparent position and size of each member of the set of size-known markers in the HMD image, and wherein calculating the orientation is further based on the apparent position, the markers positioned in the driver control area that is visible to the at least one vehicle-mounted sensor.

10. A system for dynamically orienting a presentation of a head-mounted display (HMD), comprising:

at least two image sensors including;

at least one HMD image sensor which is installed on an HMD and worn by a driver of a vehicle that captures, from a location of the driver, an HMD image of a driver control area and a part of a windshield and a scenery seen through said part of said windshield and in front of said vehicle, the scenery including a real-world object having a first apparent position from a perspective of the at least one HMD image sensor; and

at least one device image sensor of an imaging device mounted on a base fixated to said vehicle that captures a vehicle image of said scenery, the real-world object in the scenery having a second apparent position from a perspective of the at least one device image sensor; and

an analysis and calculation module comprising instructions executable by a processor of the HMD to;

perform an analysis of said HMD image and of said vehicle image to identify a difference in the first and second apparent positions of the real-world object located in said scenery and depicted both in said HMD image and in said vehicle image;

calculate, based on said difference, an orientation of said HMD in relation to said vehicle using a calculation including one or more of correlation, transformation, and translation algorithms performed between the HMD image and the vehicle image; and

adjust a presentation of data on a display of said HMD based on said orientation.

11. The system of claim 10, wherein adjusting the presentation of data on the display of the HMD based on the orientation comprises moving a location of a virtual object by an amount based on the difference between the first and second apparent positions of the real-world object responsive to determining that the data on the display comprises a virtual object tied to a real-world object inside of the vehicle.

12. The system of claim 10, wherein the HMD image includes a set of size-known markers arranged in a predetermined arrangement, and where performing the analysis of the HMD image comprises identifying an apparent position and size of each member of the set of size-known markers in the HMD image, the markers positioned in the driver control area that is visible to the device image sensor.

13. The system of claim 12, wherein identifying the apparent position and size of each member of the set of size-known markers comprises detecting the apparent position and size of each member of the set of size-known markers using a Scale-invariant feature transform (SIFT) process.

14. The system of claim 10, further comprising at least one HMD device motion detector configured to determine a current HMD device motion and at least one vehicle motion detector configured to determine a current vehicle motion, and wherein the instructions are further executable to calculate the orientation of the HMD relative to the vehicle based on a difference between the current HMD device motion and the current vehicle motion, and wherein the vehicle motion detector includes one or more of an accelerometer, a gyroscope, and a compass.

15. The system of claim 10, wherein the instructions are further executable to combine each of the HMD image and the vehicle image to create a stereoscopic image.

16. A system for dynamically orienting a presentation of a head-mounted display (HMD), comprising:

a driver control area comprising a set of size-known markers arranged in a known arrangement;

at least one HMD image sensor which is installed on an HMD worn by a driver of a vehicle that captures, when positioned on a head of the driver, an HMD image of the driver control area and a part of a windshield and a scenery seen through said part of said windshield and in front of said vehicle, and captures a marking image of said driver control area;

at least one HMD device motion detector installed in said HMD that measures a current HMD device motion;

at least one device image sensor of an imaging device mounted on a base fixated to said vehicle that captures a vehicle image of said scenery;

at least one vehicle motion detector which is installed in said vehicle that measures a current vehicle motion; and

an analysis and calculation module comprising instructions executable by a processor of at least one of the HMD and an in-vehicle computing system to;

perform an analysis to identify;

a difference in apparent position of an object located in said scenery and depicted both in said HMD image and in said vehicle image;

a difference between said current vehicle motion and said current HMD device motion; and

an apparent position and size of each member of said set of size-known markers in said marking image, the markers positioned in the driver control area that is visible to the device image sensor;

calculate, based on said differences and said apparent position, an orientation of said HMD in relation to said vehicle; and

adjust a presentation of data on a display of said HMD based on said orientation, the adjusted presentation of data maintaining a corrected position relative to the object located in the scenery based on a combination of the HMD image and the vehicle image.

17. The system of claim 16, wherein adjusting the presentation of data on the display of the HMD based on the orientation comprises moving a location of a virtual object by an amount based on the differences between the apparent position of the object and the current vehicle and HMD device motions responsive to determining that the data on the display comprises a virtual object tied to a real-world object inside of the vehicle.

18. The system of claim 16, wherein the HMD image includes a set of size-known markers arranged in a predetermined arrangement, and where performing the analysis comprises identifying an apparent position and size of each member of the set of size-known markers in the HMD image.

19. The system of claim 18, wherein identifying the apparent position and size of each member of the set of size-known markers in the HMD image comprises detecting the apparent position and size of each member of the set of size-known markers using a Scale-invariant feature transform (SIFT) process.

20. The system of claim 16, wherein the instructions are further executable to combine the HMD image and the vehicle image to create a stereoscopic image.