Virtual reality experience for a vehicle

• US 10,043,316 B2
• Filed: 08/05/2016
• Issued: 08/07/2018
• Est. Priority Date: 08/05/2016
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Donnelly, Richard, Bradley, David McAllister
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/229,923
• Publication Number:
  US 20180040162A1
• Time in Patent Office: 732 Days
• Field of Search: 701 28
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A virtual reality system provides autonomous vehicle (AV) sensor data to applications such as games and augmented reality overlays to enhance experiences for riders in the autonomous vehicle. Virtual reality headsets offer users unique and interesting experiences, but when used in a changing environment such as a moving vehicle, external stimuli can impair the virtual reality experience. AV sensors can predict these stimuli so that applications can take measures to reduce their impacts on virtual reality experiences. In addition, sensors can include cameras that send live video feeds to virtual reality devices to render improved views of the environment around the AV and of landmarks in a city. Furthermore, virtual reality devices can take advantage of the AV'"'"'s computing resources in order to offer better performance and more features to applications.

Claims

1. A virtual reality system for an autonomous vehicle (AV) comprising:

one or more processors; and

one or more memory resources storing instructions that, when executed by the one or more processors, cause the virtual reality system to;

interface with a vehicle control system to receive environmental data while the AV is being operated autonomously by the vehicle control system through an environment;

analyze the environmental data to anticipate a stimulus event, wherein the stimulus event represents probabilistic environmental stimuli on a rider in the AV;

generate virtual content using at least the environmental data and the stimulus event; and

render the virtual content for a virtual display system that presents the virtual content to the rider in the AV.

2. The virtual reality system of claim 1, further comprising instructions for:

identifying, from the environmental data, a point of interest in the environment around the AV, wherein the environmental data includes video of the environment from one or more cameras on the AV; and

retrieving content associated with the point of interest, wherein

generating the virtual content for the virtual display system uses at least the video and at least part of the retrieved content associated with the point of interest.

3. The virtual reality system of claim 2, wherein generating the virtual content includes overlaying the retrieved content onto one or more visible surfaces of the point of interest.

4. The virtual reality system of claim 2, further comprising instructions for:

tracking a head position and view direction of the rider in the AV; and

generating the virtual content based on at least the head position and view direction.

5. The virtual reality system of claim 2, further comprising instructions for:

in response to receiving an overlay selection from the rider in the AV, generating the virtual content based on the overlay selection.

6. The virtual reality system of claim 2, wherein the environmental data further includes 3D volumetric data of the environment, and the virtual reality system uses the 3D volumetric data for rendering the virtual depiction of the environment.

7. The virtual reality system of claim 1, wherein the virtual content includes (i) a realistic depiction of the environment and (ii) computer generated elements that are not depicted in the environment.

8. The virtual reality system of claim 1, wherein at least some of the environmental data is generated by acceleration sensors and at least some of the environmental data is route data for a road segment.

9. The virtual reality system of claim 8, further comprising instructions for:

receiving a starting point and a destination for the AV, wherein the road segment is a route between the starting point and the destination; and

analyzing the route data for the road segment to determine the probabilistic environmental stimuli.

10. The virtual reality system of claim 1, wherein the probabilistic environmental stimuli include one or more acceleration forces that the rider is expected to experience, including a time, direction, and magnitude of each acceleration force.

11. The virtual reality system of claim 1, further comprising instructions for:

determining that the stimulus event exceeds a programmed threshold, wherein generating the virtual content for the virtual display system is based on a plurality of stimulus events that exceed the programmed threshold.

12. The virtual reality system of claim 1, wherein the stimulus event is incorporated into a virtual reality game and the virtual content is created as part of the virtual reality game.

13. The virtual reality system of claim 1, wherein the environmental data is generated by one or more cameras, LIDAR, radar, or sonar sensors.

14. The virtual reality system of claim 1, wherein the environmental data is received, processed, or generated by the vehicle control system.

15. The virtual reality system of claim 1, wherein the virtual display system is a head-mounted display unit worn by the rider in the AV.

16. An autonomous vehicle (AV) comprising:

a sensor system to generate environmental data corresponding to an environment around the AV;

a vehicle control system to utilize the environmental data to autonomously operate the AV through the environment; and

a virtual reality system comprising;

one or more processors; and

one or more memory resources storing instructions that, when executed by the one or more processors, cause the virtual reality system to;

interface with the vehicle control system to receive environmental data while the AV is being operated autonomously by the vehicle control system through the environment;

analyze the environmental data to anticipate a stimulus event, wherein the stimulus event represents probabilistic environmental stimuli on a rider in the AV;

generate virtual content using at least the environmental data and the stimulus event; and

render the virtual content for a virtual display system that presents the virtual content to the rider in the AV.

17. The autonomous vehicle of claim 16, further comprising instructions for the virtual reality system for:

identifying, from the environmental data, a point of interest in the environment around the AV, wherein the environmental data includes video of the environment from one or more cameras on the AV; and

retrieving content associated with the point of interest, wherein

generating the virtual content for the virtual display system uses at least the video and at least part of the retrieved content associated with the point of interest.

18. The autonomous vehicle of claim 17, wherein the probabilistic environmental stimuli include one or more acceleration forces that the rider is expected to experience, including a time, direction, and magnitude of each acceleration force.

19. The autonomous vehicle of claim 16, wherein at least some of the environmental data is generated by acceleration sensors and at least some of the environmental data is route data for a road segment.

20. A non-transitory computer-readable medium storing virtual reality instructions for an autonomous vehicle (AV), wherein the instructions when executed by one or more processors, cause the one or more processors to:

interface with a vehicle control system to receive environmental data while the AV is being operated autonomously by the vehicle control system through an environment;

analyze the environmental data to anticipate a stimulus event, wherein the stimulus event represents probabilistic environmental stimuli on a rider in the AV;

generate virtual content using at least the environmental data and the stimulus event; and

render the virtual content for a virtual display system that presents the virtual content to the rider in the AV.