System and method for dynamic in-vehicle virtual reality

• US 9,715,764 B2
• Filed: 08/08/2014
• Issued: 07/25/2017
• Est. Priority Date: 10/03/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Whinnery, Joseph, Alaniz, Arthur, Murrish, Robert Wesley, Gleeson-May, Michael Eamonn
• Primary Examiner(s): Crawford, Jacinta M
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US14/455,020
• Publication Number:
  US 20150097864A1
• Time in Patent Office: 1,082 Days
• Field of Search: 345633
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: B60K 2360/55 Remote control arrangements
• CPC Class Codes: B60K 2360/589 Wireless data transfers
• CPC Class Codes: B60K 2360/77 Instrument locations other than the dashboard
• CPC Class Codes: B60K 2370/152 Displays
• CPC Class Codes: B60K 2370/155 Virtual instruments
• CPC Class Codes: B60K 2370/55 Remote controls
• CPC Class Codes: B60K 2370/589 Wireless
• CPC Class Codes: B60K 2370/77 characterised by locations other than the dashboard
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/213 Virtual instruments
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A method for in-vehicle dynamic virtual reality includes receiving vehicle data and user data from one or more portable devices, wherein the vehicle data comprises vehicle dynamics data of the vehicle. The method includes generating a virtual view based on the vehicle data, the user data and a virtual world model. The virtual world model including one or more components that define the virtual view and wherein generating the virtual view includes augmenting one or more components of the virtual world model according to at least one of the vehicle data and the user data. The method includes rendering the virtual view to an output device by controlling the output device to update display of the virtual view according to at least one of the vehicle data or the user data.

Claims

1. A computer-implemented method for in-vehicle dynamic virtual reality, comprising:

receiving vehicle dynamics data associated with a vehicle on a predetermined fixed path and receiving user data associated with a user located inside the vehicle, wherein the vehicle dynamics data and the user data are received from a portable device, wherein the portable device is an off-vehicle device including sensors mounted to a stationary structure located externally from the vehicle along the predetermined fixed path, and the sensors obtain the vehicle dynamics data and the user data externally from the vehicle, and further including determining vehicle motion data based on the vehicle dynamics data and determining user motion data representing motion of the user relative to a body of the vehicle based on the user data, the predetermined fixed path of the vehicle and the vehicle motion data;

generating a virtual view based on the vehicle dynamics data, the vehicle motion data, the predetermined fixed path, the user data, the user motion data, and a virtual world model, the virtual world model retrieved from a data store and including a group of node classes that define the virtual view, wherein generating the virtual view includes augmenting one or more node classes of the group of node classes of the virtual world model according to the vehicle dynamics data, the vehicle motion data, the predetermined fixed path, the user data, and the user motion data;

rendering the virtual view to an output device;

determining a temporal-motion rendering speed based on the vehicle motion data, the user motion data, and the predetermined fixed path, wherein the temporal-motion rendering speed is a time-based correlation between the vehicle motion data and the user motion data for the predetermined fixed path that minimizes a difference between the vehicle motion data and the user motion data; and

controlling the output device to update display of the virtual view according to the temporal-motion rendering speed.

2. The method of claim 1, wherein the off-vehicle device includes imaging units mounted to stationary structure external to the vehicle, wherein the imaging units obtain the vehicle dynamics data and the user data based on images captured by the imaging units as the vehicle passes the imaging units.

3. The method of claim 1, wherein the predetermined fixed path is determined by the off-vehicle device.

4. The method of claim 1, wherein determining the vehicle motion data comprises determining at least a velocity and an acceleration of the vehicle based on the predetermined fixed path of the vehicle.

5. The method of claim 1, wherein the off-vehicle device includes road sensors mounted to a surface of the road along the predetermined fixed path, wherein the road sensors obtain data indicating the condition of the road.

6. The method of claim 5, wherein generating the virtual view is based on the vehicle dynamics data, the predetermined fixed path, the user data, and the condition of the road.

7. A vehicle computing system for in-vehicle dynamic virtual reality, comprising:

one or more portable devices that receive and transmit vehicle dynamics data associated with a vehicle on a predetermined fixed path and user data, wherein the one or more portable devices include one or more stationary off-vehicle devices that obtain the vehicle dynamics data externally from the vehicle as the vehicle passed the one or more off-vehicle devices;

a processor operably connected to the one or more portable devices, the processor comprising;

a virtual reality data module that receives the vehicle dynamics data, the predetermined fixed path and user data from the one or more portable devices;

a dynamic virtual reality module that generates a virtual view based on the vehicle dynamics data, the predetermined fixed path, the user data and a virtual world model from the virtual reality data module, wherein the virtual world model includes a group of node classes, each node class of the group of node classes defining a property of the virtual world model, wherein generating the virtual view comprises augmenting one or more nodes of the group of node classes of the virtual world model according to the vehicle data, the predetermined fixed path and the user data; and

a rendering module that renders the virtual view generated by the dynamic reality module to an output device by controlling the output device to update display of the virtual view according to the vehicle dynamics data, the predetermined fixed path and the user data, wherein the rendering module determines vehicle motion data based on the vehicle dynamics data and the predetermined fixed path, determines user motion data representing motion of the user relative to the vehicle based on the user data and the vehicle motion data, and determines a temporal-motion rendering speed based on the vehicle motion data and the user motion data, wherein the temporal-motion rendering speed is determined as a rendering speed that minimizes a difference between the vehicle motion data and the user motion data and renders the virtual view to the output device by controlling the output device to update display of the virtual view based on the temporal-motion rendering speed thereby synchronizing the display of the virtual view in relation to the vehicle motion data and the user motion data.

8. The system of claim 7, wherein the predetermined fixed path includes predetermined navigation information about a current route of the vehicle.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a vehicle computer of a vehicle on a predetermined fixed path, causes the computer to perform the steps of:

receiving vehicle dynamics data and user data from one or more portable devices, the one or more portable devices including one or more off-vehicle devices with sensors that obtain the vehicle dynamics data;

determining vehicle motion data of the vehicle based on the vehicle dynamics data and the predetermined fixed path, wherein the vehicle motion data describes motion of the vehicle along the predetermined fixed path;

determining user motion data based on the vehicle motion data and the user data, wherein the user motion data is motion of a user located inside the vehicle with respect to the motion of the vehicle;

generating a virtual view based on the vehicle motion data, the user motion data and a virtual world model, the virtual world model including a group of node classes that define the virtual view, wherein generating the virtual view comprises augmenting one or more node classes of the group of node classes according to the vehicle motion data and the user motion data;

determining a temporal-motion rendering speed based on the vehicle motion data, the predetermined fixed path, and the user motion data, wherein the temporal-motion rendering speed is determined by calculating a difference between the user motion data and the vehicle motion data and correlating the vehicle motion data, the predetermined fixed path, and that user motion data to minimize the difference; and

rendering the virtual view to an output device by controlling the output device to update display of the virtual view according to the temporal-motion rendering speed and the virtual view.