Augmented virtual display

• US 10,482,669 B2
• Filed: 09/22/2017
• Issued: 11/19/2019
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rober, Mark B., Cohen, Sawyer I., Kurz, Daniel, Holl, Tobias, Lyon, Benjamin B., Meier, Peter Georg, Riepling, Jeffrey M., Gerhard, Holly
• Primary Examiner(s): Craddock, Robert J
• Application Number:
  US15/713,421
• Publication Number:
  US 20180089900A1
• Time in Patent Office: 788 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/211 producing three-dimensional [3D] effects, e.g. stereoscopic images
• CPC Class Codes: B60K 35/25 using haptic output
• CPC Class Codes: B60K 35/26 using acoustic output
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 10/22 including control of suspension systems
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide augmented or virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may project virtual content so that the content appears as a distant object stabilized or fixed in the external environment. The VR system may aid in productivity, as passengers may perform work while riding in the vehicle without experiencing motion sickness. In addition, the VR system may provide enhanced virtual experiences to passengers in moving vehicles by matching accelerations and motions of the vehicle to accelerations and motions in the virtual experiences.

Claims

1. A system, comprising:

a virtual reality (VR) controller in a vehicle, wherein the VR controller is configured to;

obtain inputs from one or more sources in the vehicle;

obtain display screen content from a user device of a passenger in the vehicle;

render virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;

alter the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and

send the virtual content to a VR device for display in a 3D virtual view to the passenger in the vehicle, wherein the display screen content from the user device is rendered and displayed to appear to the passenger to be at a fixed distance in front of the vehicle.

2. The system as recited in claim 1, wherein the VR device includes:

a head-mounted device (HMD) worn by the passenger that displays the virtual content for viewing by the passenger;

or

a projector that projects the virtual content to a window of the vehicle for viewing by the passenger.

3. The system as recited in claim 1, wherein the user device comprises one or more of a notebook computer, a mobile device, a smartphone, or a tablet.

4. The system as recited in claim 1, wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

5. The system as recited in claim 1, wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues that indicate motion of the vehicle in the virtual environment to the passenger.

6. The system as recited in claim 5, wherein the VR controller is further configured to:

determine that the passenger is prone to or is exhibiting signs of motion sickness; and

upon determining that the passenger is prone to or is exhibiting signs of motion sickness, alter the virtual environment according to one or more accommodations to mitigate motion sickness.

7. The system as recited in claim 6, wherein the VR controller is configured to determine that the passenger is prone to motion sickness according to the preference data for the passenger or input from the passenger.

8. The system as recited in claim 6, wherein the VR controller is configured to determine that the passenger is exhibiting signs of motion sickness according to passenger sensor data obtained from one or more sensors that monitor the passenger.

9. The system as recited in claim 6, wherein the one or more accommodations include:

displaying a ground plane beneath the passenger in the virtual reality view;

reducing a number of visual cues that indicate motion of the vehicle in the virtual environment;

slowing the visual cues that indicate motion of the vehicle in the virtual environment so that there is a 1;

1 ratio between motion in the virtual environment and motion in the real environment;

providing visual, audio, or haptic indications of upcoming vehicle maneuvers;

reducing the virtual content displayed in the virtual reality view to provide a sparse visual environment;

placing an anchored object in the virtual environment that remains fixed relative to the real world;

providing a visual stimulus at stops of the vehicle to counter visual motion aftereffects;

or

providing audio output that blocks real world noise.

10. The system as recited in claim 1, wherein the one or more sources include:

one or more vehicle sensors, wherein the inputs from the vehicle sensors include one or more of depth map information from external vehicle sensors, localization information for the vehicle, video from one or more cameras on the vehicle, or motion and orientation information for the vehicle from an inertial-measurement unit (IMU) of the vehicle; and

one or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

11. The system as recited in claim 1, wherein the one or more sources include sources external to the vehicle that are accessed through wireless connections, wherein inputs from the external sources include one or more of world map data, 3D models of a local environment, information about objects or features in the local environment, video streams, or audio.

12. The system as recited in claim 1, wherein the one or more sources include an inertial-measurement unit (IMU) of the VR device, wherein inputs from the IMU include motion and orientation information for the passenger'"'"'s head.

13. The system as recited in claim 1, wherein the vehicle is an autonomous vehicle.

14. The system as recited in claim 1, wherein, in rendering the virtual content, the VR controller synchronizes motions and accelerations of the virtual content with motions and accelerations of the vehicle based on the inputs from the one or more sources in the vehicle.

15. A method, comprising:

obtaining, by a virtual reality (VR) system in a vehicle, inputs from one or more sources in the vehicle;

obtaining, by the VR system, display screen content from a user device of a passenger in the vehicle;

rendering, by the VR system, virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;

altering the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and

displaying, by the VR system, the virtual content in a 3D virtual view to a passenger in the vehicle, wherein the display screen content appears to the passenger to be at a fixed distance in front of the vehicle.

16. The method as recited in claim 15, wherein the VR system includes a VR controller and a VR device, wherein the VR device is one of a head-mounted device (HMD) configured to be worn by the passenger and to display or project the virtual content for viewing by the passenger or a projector configured to project the virtual content to a window of the vehicle for viewing by the passenger.

17. The method as recited in claim 15, wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

18. The method as recited in claim 15, wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues to indicate motion of the vehicle in the virtual environment to the passenger.

19. The method as recited in claim 15, further comprising:

determining that the passenger is prone to or is exhibiting signs of motion sickness, wherein altering the virtual content according to the one or more visual or audio accommodations to mitigate motion sickness is performed upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

20. The method as recited in claim 15, wherein rendering the virtual content comprises synchronizing motions and accelerations of the virtual content with motions and accelerations of the vehicle based on the inputs from the one or more sources in the vehicle.