Immersive virtual display

• US 10,643,391 B2
• Filed: 09/22/2017
• Issued: 05/05/2020
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rober, Mark B., Cohen, Sawyer I., Kurz, Daniel, Holl, Tobias, Lyon, Benjamin B., Meier, Peter Georg, Riepling, Jeffrey M., Gerhard, Holly
• Primary Examiner(s): Liu, Zhengxi
• Application Number:
  US15/713,428
• Publication Number:
  US 20180089901A1
• Time in Patent Office: 956 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/211 producing three-dimensional [3D] effects, e.g. stereoscopic images
• CPC Class Codes: B60K 35/25 using haptic output
• CPC Class Codes: B60K 35/26 using acoustic output
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 10/22 including control of suspension systems
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may provide immersive VR experiences by replacing the view of the real world with virtual environments. Active vehicle systems and/or vehicle control systems may be integrated with the VR system to provide physical effects with the virtual experiences. The virtual environments may be altered to accommodate a passenger upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

Claims

1. A system, comprising:

a virtual reality (VR) controller in a vehicle, wherein the VR controller comprises one or more processors and memory configured to;

obtain inputs from one or more sources in a vehicle following an actual route in a real environment at a location;

select a route in another real location, wherein selection is based on comparing turns and curves of the route in the other real location to turns and curves of the actual route, and wherein the route in the other location is selected based at least in part on the turns and curves of the route in the other real location at least partially matching the turns and curves of the actual route;

render frames including virtual content having a virtual environment comprising a simulation of the selected route in the other real location, wherein the virtual environment includes a virtual view that is rendered based on the selected route in the other real location, and wherein, in rendering the frames, the VR controller synchronizes motions and accelerations of the virtual content with motions and accelerations of the vehicle traveling along the actual route based on the inputs from the one or more sources in the vehicle;

send the frames to a VR device for display to a passenger in the vehicle, wherein the frames are rendered and displayed to provide the virtual view of the virtual environment that replaces a view of the actual route in the real environment from the vehicle via the VR device.

2. The system as recited in claim 1, wherein the VR device includes a head-mounted device (HMD) that displays the frames to the passenger.

3. The system as recited in claim 1, wherein the view of the virtual environment includes visual cues to indicate motion of the vehicle to the passenger.

4. The system as recited in claim 1, wherein the one or more sources include:

one or more vehicle sensors, wherein the inputs from the vehicle sensors include one or more of depth map information from external vehicle sensors, localization information for the vehicle, video from one or more cameras on the vehicle, or motion and orientation information for the vehicle from an inertial-measurement unit (IMU) of the vehicle; and

one or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

5. The system as recited in claim 1, wherein the one or more sources include sources external to the vehicle that are accessed through wireless connections, wherein inputs from the external sources include one or more of world map data, 3D models of a local environment, information about objects or features in the local environment, video streams, or audio.

6. The system as recited in claim 1, wherein the one or more sources include an inertial-measurement unit (IMU) of the VR device, wherein inputs from the IMU include motion and orientation information for the passenger'"'"'s head.

7. The system as recited in claim 1, wherein the VR controller is further configured to:

determine that the selected route does not exactly match the actual route; and

modify the simulation of the selected route based on differences between the actual route and the selected route.

8. The system as recited in claim 1, wherein the VR controller is further configured to provide audio to the passenger through an audio system of the vehicle or through an audio system of the VR device.

9. The system as recited in claim 1, wherein the VR controller is further configured to signal vehicle control systems to cause the vehicle control systems to react within constraints to provide physical effects to the passenger, wherein the vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

10. The system as recited in claim 1, wherein the VR controller is further configured to signal an HVAC system of the vehicle to cause the HVAC system to change one or more of fan speed, temperature, or direction to provide physical effects to the passenger.

11. The system as recited in claim 1, wherein the VR controller is further configured to signal an active seat to cause the seat to move or tilt to provide physical effects to the passenger.

12. The system as recited in claim 1, wherein the VR controller is further configured to receive interactive inputs from the passenger and modify rendering of the frames including virtual reality content in response to the interactive inputs, wherein the interactive inputs include one or more of gestures, voice commands, or inputs to an input device.

13. The system as recited in claim 1, wherein the VR controller is further configured to render avatars of two or more passengers in the vehicle in the virtual environment, wherein the two or more passengers participate in the virtual environment via respective VR devices.

14. The system as recited in claim 1, wherein the VR controller is further configured to:

determine that the passenger is prone to or is exhibiting signs of motion sickness; and

upon determining that the passenger is prone to or is exhibiting signs of motion sickness, alter the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger, wherein the one or more accommodations include;

displaying a ground plane beneath the passenger in the view of the virtual environment;

reducing visual cues that indicate motion of the vehicle in the virtual environment;

slowing the visual cues that indicate motion of the vehicle in the virtual environment so that there is a 1;

1 ratio between motion in the virtual environment and motion in the real environment;

providing visual, audio, or haptic indications of upcoming vehicle maneuvers;

reducing the virtual content to provide a sparse visual environment;

placing an anchored object in the virtual environment that remains fixed relative to the real environment;

providing a visual stimulus at stops of the vehicle to counter visual motion aftereffects;

or

providing audio output that blocks real world noise.

15. A method, comprising:

obtaining, by a virtual reality (VR) system in a vehicle, inputs from one or more sources in the vehicle following an actual route in a location;

selecting, by the VR system, a route in another real location, wherein selecting is based on comparing turns and curves of the route in the other real location to turns and curves of the actual route, and wherein the route in the other real location is selected based at least in part on the turns and curves of the route in the other real location at least partially matching the turns and curves of the actual route;

rendering, by the VR system, frames including virtual content having a virtual environment comprising a simulation of the route in the other real location, wherein the virtual environment includes a virtual view that is rendered based on the route in the other real location, and wherein, in rendering the frames, motions and accelerations of the virtual content are synchronized with motions and accelerations of the vehicle traveling along the actual route based on the inputs from the one or more sources in the vehicle; and

displaying, by the VR system, the frames to at least one passenger in the vehicle, wherein the displayed frames provide the virtual view of the virtual environment that replaces a view of the actual route in reality from the vehicle.

16. The method as recited in claim 15, wherein the VR system includes a VR controller and a VR device, wherein the VR device is one of a head-mounted device (HMD) worn by the passenger that displays the frames for viewing by the passenger or a projector that projects the frames to a window of the vehicle for viewing by the passenger, the method further comprising providing audio to the passenger through an audio system of the vehicle or through an audio system of the VR device.

17. The method as recited in claim 15, wherein the view of the virtual environment includes visual cues to indicate motion of the vehicle to the passenger.

18. The method as recited in claim 15, further comprising signaling one or more vehicle systems to provide physical effects to the passenger that correspond to the view of the virtual environment displayed to the passenger.

19. The method as recited in claim 15, further comprising receiving interactive inputs from the passenger and modifying rendering of the frames including virtual reality content in response to the interactive inputs.

20. The method as recited in claim 15, further comprising altering the virtual environment to accommodate the passenger according to preference data of the passenger upon determining that the passenger is prone to or is exhibiting signs of motion sickness.