Virtual reality experience for a vehicle

• US 9,922,466 B2
• Filed: 08/05/2016
• Issued: 03/20/2018
• Est. Priority Date: 08/05/2016
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Donnelly, Richard, Bradley, David McAllister
• Primary Examiner(s): CHOW, JEFFREY J
• Application Number:
  US15/230,053
• Publication Number:
  US 20180040163A1
• Time in Patent Office: 592 Days
• CPC Class Codes: B60R 1/22 for viewing an area outside the vehicle, e.g. the exterior of the vehicle
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A virtual reality system is disclosed that provides autonomous vehicle (AV) sensor data to applications such as games and augmented reality overlays to enhance experiences for riders in the autonomous vehicle. Virtual reality headsets offer users unique and interesting experiences, but when used in a changing environment such as a moving vehicle, external stimuli can impair the virtual reality experience. AV sensors can predict these stimuli so that applications can take measures to reduce their impacts on virtual reality experiences. In addition, sensors can include cameras that send live video feeds to virtual reality devices to render improved views of the environment around the AV and of landmarks in a city. Furthermore, virtual reality devices can take advantage of the AV'"'"'s computing resources in order to offer better performance and more features to applications.

Claims

1. An autonomous vehicle (AV) comprising:

a sensor system to generate sensor data corresponding to a situational environment of the AV;

a data processing system to determine that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;

a control system to utilize the sensor data to autonomously control the AV along a current route;

one or more computing resources shared between a virtual reality system and other systems of the AV, including at least one of the sensor system, the data processing system, and the control system; and

the virtual reality system to;

provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and

manage utilization of the one or more computing resources between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.

2. The autonomous vehicle of claim 1, wherein the virtual reality application accesses the sensor data through an application programming interface provided by the virtual reality system.

3. The autonomous vehicle of claim 1, wherein the sensor system tracks a head position and view direction of the rider in the AV and provides the head position and view direction to the virtual reality application.

4. The autonomous vehicle of claim 1, wherein the sensor system tracks a hand position and body position for one or more riders in the AV and provides the hand position and the body position for each of the riders to the virtual reality application.

5. The autonomous vehicle of claim 1, wherein the virtual reality system retrieves content from one or more databases over a network based on the sensor data and provides the retrieved content to the virtual reality application.

6. The autonomous vehicle of claim 1, wherein the data processing system retrieves route data corresponding to the current route from one or more databases over a network, and the virtual reality system provides the route data to the virtual reality application.

7. The autonomous vehicle of claim 1, wherein the sensor system tracks one or more possessions associated with the rider in the AV and warns the rider when a different rider in the AV accesses one of the possessions.

8. The autonomous vehicle of claim 1, wherein the virtual reality system provides a 3D model of the interior of the AV to the virtual reality application.

9. The autonomous vehicle of claim 8, wherein the 3D model of the interior of the AV provided to the virtual reality application includes a representation of haptic textures of surfaces in the interior of the AV.

10. The autonomous vehicle of claim 1, further comprising a signal emitter to communicate with one or more remote sensors in the situational environment, wherein the one or more remote sensors provide the AV with further sensor data.

11. A virtual reality system for an autonomous vehicle (AV) comprising:

one or more computing resources, including one or more processors, shared between the virtual reality system and other systems of the AV, including at least one of a sensor system, a data processing system, and a control system for the AV that utilizes the sensor data to autonomously control the AV along a current route; and

one or more memory resources storing instructions that, when executed by the one or more processors, cause the virtual reality system to;

retrieve, from the data processing system of the AV, sensor data corresponding to a situational environment of the AV, wherein the data processing system determines that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;

provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and

manage utilization of the one or more computing resources between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.

12. The virtual reality system of claim 11, wherein the virtual reality application accesses the sensor data through an application programming interface provided by the virtual reality system.

13. The virtual reality system of claim 11, wherein the sensor system tracks a head position and view direction of the rider in the AV and provides the head position and view direction to the virtual reality application.

14. The virtual reality system of claim 11, wherein the sensor system tracks a hand position and body position of the rider in the AV and provides the hand position and the body position to the virtual reality application.

15. The virtual reality system of claim 11, wherein the virtual reality system retrieves content from one or more databases over a network based on the sensor data and provides the retrieved content to the virtual reality application.

16. The virtual reality system of claim 11, wherein the data processing system retrieves route data corresponding to the current route from one or more databases over a network, and the virtual reality system provides the route data to the virtual reality application.

17. A non-transitory computer-readable medium storing instructions for a virtual reality system of an autonomous vehicle (AV), wherein the instructions when executed by one or more processors, cause the one or more processors to:

retrieve, from a data processing system of the AV, sensor data corresponding to a situational environment of the AV, wherein the data processing system determines that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;

provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and

manage utilization of one or more computing resources, shared between the virtual reality system and other systems of the AV, including at least one of a sensor system, the data processing system, and a control system for the AV that utilizes the sensor data to autonomously control the AV along a current route, between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.