System and method for remote virtual reality control of movable vehicle partitions

• US 10,249,088 B2
• Filed: 11/20/2014
• Issued: 04/02/2019
• Est. Priority Date: 11/20/2014
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Whinnery, Joseph
• Primary Examiner(s): Han, Charles J
• Application Number:
  US14/548,596
• Publication Number:
  US 20160147233A1
• Time in Patent Office: 1,594 Days
• Field of Search: 701 2
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/55 Remote control arrangements
• CPC Class Codes: B60K 2360/573 Mobile devices controlling vehicle functions
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method for remote virtual reality control of movable vehicle partitions includes displaying a graphic model of at least a portion of a vehicle on an output device. The output device is located remotely from the vehicle and the vehicle has one or more movable vehicle partitions. The method includes processing images received from one or more imaging devices, the images including gestures relative to the graphic model. The method includes identifying one or more vehicle commands based on the gestures relative to the graphic model. The vehicle commands define control of the one or more movable vehicle partitions. The method further includes executing the vehicle commands at the vehicle.

Claims

1. A computer-implemented method for remote virtual reality control of movable vehicle partitions, comprising:

displaying a three-dimensional virtual reality model of at least a portion of a vehicle on a display of an output device, the output device located remotely from the vehicle, the vehicle having one or more movable vehicle partitions;

receiving images that capture gestures made in relation to the three-dimensional virtual reality model displayed on the output device from one or more imaging devices of the output device;

identifying one or more vehicle commands in a sequence based on the images that capture the gestures made in relation to the three-dimensional virtual reality model, the one or more vehicle commands defining control of the one or more movable vehicle partitions, and storing the one or more vehicle commands in the sequence in a memory of the output device;

receiving real-time video feed from one or more imaging devices located adjacent to the vehicle;

wherein the display of the output device includes a first user interface and a second user interface, displaying the three-dimensional virtual reality model on the first interface and streaming the real-time video feed of the vehicle to the second interface;

and upon determining the output device is located within a predetermined range of the vehicle based upon receiving an authentication signal at the output device from the vehicle when the output device is located within the predetermined range, transmitting the one or more vehicle commands in the sequence from the memory of the output device to the vehicle and executing the one or more vehicle commands at the vehicle.

2. The computer-implemented method of claim 1, comprising updating the display of the three-dimensional virtual reality model based on the gestures made in relation to the three-dimensional virtual reality model.

3. The computer-implemented method of claim 1, comprising streaming a real-time video feed of the vehicle to the output device, the video feed received from one or more imaging devices located next to the vehicle.

4. The computer-implemented method of claim 3, comprising streaming a real-time audio feed to the output device corresponding to the real-time video feed of the vehicle.

5. The computer-implemented method of claim 1, comprising generating a real-time virtual reality model of the vehicle based on sensor data received from the vehicle and displaying the real-time virtual reality model of the vehicle on the output device simultaneously with the three-dimensional virtual reality model.

6. A system for remote virtual reality control of movable vehicle partitions, comprising:

a vehicle having a controller, one or more movable vehicle partitions, and a passive entry system, the controller operably connected for computer communication to an output device;

the output device, located remotely from the vehicle, the output device having a display, wherein the output device displays a three-dimensional virtual reality model of at least a portion of the vehicle on the display, wherein the output device includes a processor and wherein the processor includes;

a gesture module that processes images received from one or more imaging devices, the images including gestures made to manipulate the three-dimensional virtual reality model displayed on the output device;

and a vehicle command module that identifies a plurality of vehicle commands and a sequence of the plurality of vehicle commands based on the gestures made to manipulate the three-dimensional virtual reality model and stores the plurality of vehicle commands and the sequence of the plurality of vehicle commands in a memory of the output device, wherein the plurality of vehicle commands define control of the one or more movable vehicle partitions according to the sequence,

wherein upon the passive entry system authenticating the output device when the passive entry system determines the output device is within a predetermined range of the vehicle, the vehicle command module transmits the plurality of vehicle commands and the sequence of the plurality of vehicle commands from the memory of the output device to the controller for execution at the vehicle according to the sequence;

wherein the display of the output device includes a first user interface and a second user interface; and

wherein the gesture module displays the three-dimensional virtual reality model on the first interface and the gesture module streams a real-time video feed of the vehicle to the second interface, the video feed received from one or more imaging devices located next to the vehicle.

7. The system of claim 6, wherein the gesture module generates a three-dimensional real-time virtual reality model of the vehicle based on sensor data received from the vehicle and displays the three-dimensional real-time virtual reality model of the vehicle on the second interface.

8. The system of claim 6, wherein the output device includes a database of one or more three-dimensional virtual reality models, each model associated with a vehicle and a user associated with the vehicle.

9. The system of claim 8, wherein the vehicle command module stores the plurality of vehicle commands in the database associated with the user and the vehicle associated with the three-dimensional virtual reality model displayed on the output device.

10. The system of claim 6, wherein the output device updates the display of the three-dimensional virtual reality model based on the gestures made to manipulate the three-dimensional virtual reality model.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, causes the computer to perform a method comprising:

displaying a three-dimensional virtual reality model of at least a portion of a vehicle on an output device, the output device located remotely from the vehicle, the vehicle having one or more movable vehicle partitions;

generating a real-time virtual reality model of the vehicle based on data received at the output device from vehicle sensors of the vehicle;

displaying the real-time virtual reality model of the vehicle on a display of the output device simultaneously with the three-dimensional virtual reality model;

receiving images from one or more imaging devices, the images including gestures made relative to the three-dimensional virtual reality model;

identifying a plurality of one or more vehicle commands in a sequence based on the images including the gestures made to interact with the three-dimensional virtual reality model, the one or more vehicle commands defining control of the one or more movable vehicle partitions;

receiving real-time video feed from one or more imaging devices located adjacent to the vehicle;

wherein the display of the output device includes a first user interface and a second user interface, displaying the three-dimensional virtual reality model on the first interface and streaming the real-time video feed of the vehicle to the second interface; and

upon determining the output device is located within a predetermined range of the vehicle, verifying the one or more vehicle commands for execution based on sensor data received from the vehicle and predetermined-rules for controlling movable partitions of the vehicle, and transmitting the one or more vehicle commands that have been verified to the vehicle for execution.

12. The computer-implemented method of claim 11, further comprising updating the display of the three-dimensional virtual reality model based on the gestures made to interact with the three-dimensional virtual reality model.