Virtual reality headset with relative motion head tracker

• US 9,459,692 B1
• Filed: 03/29/2016
• Issued: 10/04/2016
• Est. Priority Date: 03/29/2016
• Status: Expired due to Fees

• Current Assignee: Adam Li
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Li, Adam
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/084,327
• Time in Patent Office: 189 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/176 Camera images
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/48 Sensors
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30268 : Vehicle interior
• CPC Class Codes: G06T 7/248 : involving reference images or patches

Abstract

A virtual reality headset that tracks motion of a user relative to a moving vehicle in which the user is located. The system updates the headset'"'"'s display based on relative motion, rather than on absolute motion due to vehicle movement. Data from sensors on the headset, the vehicle, or both is analyzed to determine the pose (orientation, position, or both) of the headset relative to the vehicle. Sensors may include headset-mounted cameras that observe the vehicle'"'"'s interior from the user'"'"'s viewpoint. Analysis of camera images may use optical flow or it may compare images to reference images captured from known poses. The system may also combine data from headset sensors and vehicle sensors to calculate the relative pose. Headset-mounted cameras may function both as relative pose sensors for virtual reality, and as image capture devices for display of the real environment on the headset.

Claims

1. A virtual reality headset with a relative motion head tracker, comprising

a headset coupled to a head of a user, wherein said user is located in or on a vehicle that may move;

a display coupled to said headset and viewable by said user; and

,

a relative motion head tracker comprising

one or more sensors that generate sensor data, wherein said sensor data measures one or more characteristics that vary based on one or both of a pose of said headset and a pose of said vehicle; and

,

a processor coupled to said one or more sensors and to said display;

wherein said processor is configured to

receive said sensor data;

calculate a pose of said headset relative to said vehicle from said sensor data;

generate, retrieve, select, or modify a display image based on said pose of said headset relative to said vehicle; and

,

transmit said display image to said display;

wherein said one or more characteristics comprise an acceleration of said vehicle on at least one axis; and

,

wherein said processor is further configured to

when said acceleration of said vehicle is outside a normal range, generate, retrieve, select, or modify said display image based on a last pose of said headset relative to said vehicle that was calculated when said acceleration of said vehicle was in said normal range.

2. The virtual reality headset with a relative motion head tracker of claim 1, wherein said vehicle comprises a car, a truck, a bus, a motorcycle, a sidecar, a carriage, a train, a subway, a trolley, a tram, an airplane, a balloon, a dirigible, a hang glider, a space station, a space ship, a rocket, a boat, or a submarine.

3. The virtual reality headset with a relative motion head tracker of claim 1, wherein

said one or more sensors comprise a camera coupled to said headset and oriented to capture a time series of images of at least a portion of said vehicle;

and,

said calculate a pose of said headset relative to said vehicle from said sensor data comprises

analyze said time series of images to determine said pose of said headset relative to said vehicle.

4. The virtual reality headset with a relative motion head tracker of claim 3, wherein

said analyze said time series of images comprises

calculate an optical flow of said time series of images;

and,

derive an egomotion of said camera from said optical flow.

5. The virtual reality headset with a relative motion head tracker of claim 3, wherein

said analyze said time series of images comprises

compare an image of said time series of images to one or more reference images of said vehicle, each associated with a reference pose of said headset relative to said vehicle;

calculate a pose transformation that aligns said image with a selected reference image of said one or more reference images; and

,

calculate said pose of said headset relative to said vehicle by applying said pose transformation to said reference pose associated with said selected reference image.

6. The virtual reality headset with a relative motion head tracker of claim 5, further comprising

one or more reference pose sensors configured to measure said reference pose associated with each of said one or more reference images;

and,

a reference image acquisition system configured to

capture said one or more reference images using said camera;

for each of said one or more reference images, obtain a corresponding reference pose from said one or more reference pose sensors; and

,

store each of said one or more reference images and its corresponding reference pose in a reference image database.

7. The virtual reality headset with a relative motion head tracker of claim 1, wherein

said one or more sensors comprise

one or more vehicle pose sensors that measure a pose of said vehicle relative to an external reference frame;

and,

one or more headset pose sensors that measure a pose of said headset relative to said external reference frame; and

,

said processor is configured to calculate said pose of said headset relative to said vehicle based on said pose of said headset relative to said external reference frame and on said pose of said vehicle relative to said external reference frame.

8. The virtual reality headset with a relative motion head tracker of claim 7, wherein

said external reference frame is an inertial reference frame;

and,

said one or more headset pose sensors comprise one or more of

a headset accelerometer;

a headset gyroscope;

a headset magnetometer; and

,

a headset GPS.

9. The virtual reality headset with a relative motion head tracker of claim 7, wherein said one or more vehicle pose sensors comprise one or more of

a vehicle accelerometer;

a vehicle gyroscope;

a vehicle magnetometer;

a vehicle GPS; and

,

a vehicle camera coupled to said vehicle and oriented to capture images of a surroundings of said vehicle.

10. The virtual reality headset with a relative motion head tracker of claim 3, wherein

said processor is further configured to operate in a virtual mode or in a real mode;

and,

when said processor is operating in said real mode, said processor transmits at least a portion of said time series of images to said display coupled to said headset.

11. The virtual reality headset with a relative motion head tracker of claim 10, wherein

said processor is further configured to operate in an augmented reality mode;

and,

when said processor is operating in said augmented reality mode, said processor

modifies said at least a portion of said time series of images, forming augmented images; and

,

transmits said augmented images to said display coupled to said headset.

12. A virtual reality headset with a relative motion head tracker, comprising

a headset coupled to a head of a user, wherein said user is located in or on a vehicle that may move;

a display coupled to said headset and viewable by said user; and

,

a relative motion head tracker comprising

one or more sensors that generate sensor data, wherein said sensor data measures one or more characteristics that vary based on one or both of a pose of said headset and a pose of said vehicle; and

,

a processor coupled to said one or more sensors and to said display;

wherein said processor is configured to

receive said sensor data;

calculate a pose of said headset relative to said vehicle from said sensor data;

generate, retrieve, select, or modify a display image based on said pose of said headset relative to said vehicle; and

,

transmit said display image to said display;

wherein said one or more sensors comprise a camera coupled to said headset and oriented to capture a time series of images of at least a portion of said vehicle;

wherein said calculate a pose of said headset relative to said vehicle from said sensor data comprises

analyze said time series of images to determine said pose of said headset relative to said vehicle; and

,

wherein said analyze said time series of images comprises

compare an image of said time series of images to one or more reference images of said vehicle, each associated with a reference pose of said headset relative to said vehicle;

calculate a pose transformation that aligns said image with a selected reference image of said one or more reference images; and

,

calculate said pose of said headset relative to said vehicle by applying said pose transformation to said reference pose associated with said selected reference image.

13. The virtual reality headset with a relative motion head tracker of claim 12, wherein said vehicle comprises a car, a truck, a bus, a motorcycle, a sidecar, a carriage, a train, a subway, a trolley, a tram, an airplane, a balloon, a dirigible, a hang glider, a space station, a space ship, a rocket, a boat, or a submarine.

14. The virtual reality headset with a relative motion head tracker of claim 12, wherein said analyze said time series of images comprises

calculate an optical flow of said time series of images;

and,

derive an egomotion of said camera from said optical flow.

15. The virtual reality headset with a relative motion head tracker of claim 12, further comprising

one or more reference pose sensors configured to measure said reference pose associated with each of said one or more reference images;

and,

a reference image acquisition system configured to

capture said one or more reference images using said camera;

for each of said one or more reference images, obtain a corresponding reference pose from said one or more reference pose sensors; and

,

store each of said one or more reference images and its corresponding reference pose in a reference image database.

16. The virtual reality headset with a relative motion head tracker of claim 12, wherein

said one or more sensors comprise

one or more vehicle pose sensors that measure a pose of said vehicle relative to an external reference frame;

and,

one or more headset pose sensors that measure a pose of said headset relative to said external reference frame; and

,

said processor is configured to calculate said pose of said headset relative to said vehicle based on said pose of said headset relative to said external reference frame and on said pose of said vehicle relative to said external reference frame.

17. The virtual reality headset with a relative motion head tracker of claim 16, wherein

said external reference frame is an inertial reference frame;

and,

said one or more headset pose sensors comprise one or more of

a headset accelerometer;

a headset gyroscope;

a headset magnetometer; and

,

a headset GPS.

18. The virtual reality headset with a relative motion head tracker of claim 16, wherein

said one or more vehicle pose sensors comprise one or more of

a vehicle accelerometer;

a vehicle gyroscope;

a vehicle magnetometer;

a vehicle GPS; and

,

a vehicle camera coupled to said vehicle and oriented to capture images of a surroundings of said vehicle.

19. The virtual reality headset with a relative motion head tracker of claim 12, wherein

said processor is further configured to operate in a virtual mode or in a real mode;

and,

when said processor is operating in said real mode, said processor transmits at least a portion of said time series of images to said display coupled to said headset.

20. The virtual reality headset with a relative motion head tracker of claim 19, wherein

said processor is further configured to operate in an augmented reality mode;

and,

when said processor is operating in said augmented reality mode, said processor

modifies said at least a portion of said time series of images, forming augmented images; and

,

transmits said augmented images to said display coupled to said headset.

21. The virtual reality headset with a relative motion head tracker of claim 12, wherein

said one or more characteristics comprise an acceleration of said vehicle on at least one axis;

and,

said processor is further configured to

when said acceleration of said vehicle is outside a normal range, generate, retrieve, select, or modify said display image based on a last pose of said headset relative to said vehicle that was calculated when said acceleration of said vehicle was in said normal range.

22. A virtual reality headset with a relative motion head tracker, comprising

a headset coupled to a head of a user, wherein said user is located in or on a vehicle that may move;

a display coupled to said headset and viewable by said user;

a relative motion head tracker comprising

one or more sensors that generate sensor data, wherein said sensor data measures one or more characteristics that vary based on one or both of a pose of said headset and a pose of said vehicle; and

,

a processor coupled to said one or more sensors and to said display;

wherein said processor is configured to

receive said sensor data;

calculate a pose of said headset relative to said vehicle from said sensor data;

generate, retrieve, select, or modify a display image based on said pose of said headset relative to said vehicle; and

,

transmit said display image to said display;

wherein said one or more sensors comprise a camera coupled to said headset and oriented to capture a time series of images of at least a portion of said vehicle;

wherein said calculate a pose of said headset relative to said vehicle from said sensor data comprises

analyze said time series of images to determine said pose of said headset relative to said vehicle;

wherein said analyze said time series of images comprises

compare an image of said time series of images to one or more reference images of said vehicle, each associated with a reference pose of said headset relative to said vehicle;

calculate a pose transformation that aligns said image with a selected reference image of said one or more reference images; and

,

calculate said pose of said headset relative to said vehicle by applying said pose transformation to said reference pose associated with said selected reference image;

one or more reference pose sensors configured to measure said reference pose associated with each of said one or more reference images; and

,

a reference image acquisition system configured to

capture said one or more reference images using said camera;

for each of said one or more reference images, obtain a corresponding reference pose from said one or more reference pose sensors;

store each of said one or more reference images and its corresponding reference pose in a reference image database.