SYSTEM AND METHOD FOR PROVIDING AN AUGMENTED REALITY VEHICLE INTERFACE

• US 20150202962A1
• Filed: 01/21/2014
• Published: 07/23/2015
• Est. Priority Date: 01/21/2014
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Habashima, Yoshiyuki, Kurosawa, Fuminobu, Alaniz, Arthur, Gleeson-May, Michael
• Granted Patent: US 9,550,419 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/122 Instrument input devices with reconfigurable control functions, e.g. reconfigurable menus
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/347 Optical elements for superposition of display information
• CPC Class Codes: B60K 2360/55 Remote control arrangements
• CPC Class Codes: B60K 2360/573 Mobile devices controlling vehicle functions
• CPC Class Codes: B60K 2360/589 Wireless data transfers
• CPC Class Codes: B60K 2370/122 with reconfigurable control functions, e.g. reconfigurable menus
• CPC Class Codes: B60K 2370/177 Augmented reality
• CPC Class Codes: B60K 2370/21 using cameras
• CPC Class Codes: B60K 2370/55 Remote controls
• CPC Class Codes: B60K 2370/573 Mobile devices controlling vehicle functions
• CPC Class Codes: B60K 2370/589 Wireless
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60K 37/06 of controls, e.g. controls knobs
• CPC Class Codes: G02B 2027/0138 : comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04817 : using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08G 1/00 : Traffic control systems for road vehicles arrangement of road signs or traffic signals E01F9/00 ; automatic vehicle control B62D
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages

Abstract

A method and system for providing an augmented reality vehicle interface. The method and system include providing an augmented reality user interface. The method and system additionally include receiving an image of a vehicle with an image capturing device. The method and system additionally include identifying a user classification category of the user that is capturing an image of the vehicle. Additionally, the method and system include presenting an augmented reality image of the vehicle by overlaying one or more virtual user interface objects on the points of interest. The method and system also includes controlling vehicle features via the one or more virtual user interface objects.

Claims

1. A method for providing an augmented reality vehicle interface comprising:

receiving an image of a vehicle with an image capturing device;

identifying points of interest within portions of the image of the vehicle that correspond with vehicle features;

identifying a user classification category of the user that is capturing an image of the vehicle;

presenting an augmented reality image of the vehicle by overlaying one or more virtual user interface objects on the points of interest; and

controlling vehicle features via the one or more virtual user interface objects.

2. The method of claim 1, wherein identifying points of interest within portions of the image of the vehicle that correspond with vehicle features includes determining the directional orientation and positional location of the image capturing device with respect to the vehicle and identifying vehicle features being captured by the image capturing device based on the direction and position of the image capturing device with respect to the vehicle.

3. The method of claim 1, wherein identifying points of interest within portions of the image of the vehicle that correspond with vehicle features includes identifying the location of a predetermined marker as a focal point within the image and identifying points of interests based on the location of the predetermined marker by determining the specific orientation of the image capturing device.

4. The method of claim 1, wherein identifying a user classification category of the user that is capturing the image of the vehicle includes determining if a driver or passenger is operating the image capturing device while capturing the image of the vehicle.

5. The method of claim 4, wherein determining if a driver or passenger is operating the image capturing device while capturing the image of the vehicle includes determining the cardinal and ordinal position of the image capturing device with respect to the vehicle while the user is capturing the image of the vehicle.

6. The method of claim 5, wherein presenting an augmented reality image of the vehicle includes providing one or more virtual user interface objects shown on an augmented reality user interface that includes a plurality of vehicle feature interface icons that correspond to the vehicle features associated with one or both of the driver or passenger that is operating the image capturing device when capturing the image of the vehicle.

7. The method of claim 6, wherein presenting an augmented reality image of the vehicle includes providing one or more virtual user interface objects shown on an augmented reality user interface that includes a plurality of vehicle feature settings menus icons on a vehicle feature settings menu that are utilized to determine the settings associated with the vehicle features associated with one or both of the driver or passenger that is operating the image capturing device when capturing the image of the vehicle.

8. The method of claim 7, wherein presenting an augmented reality image of the vehicle includes providing one or more virtual user interface objects shown on the augmented reality user interface that includes a plurality of vehicle feature settings sub-menu icons on a vehicle feature settings sub-menu that are utilized to manipulate the settings associated with the vehicle features associated with one or both of the driver or passenger that is operating the image capturing device when capturing the image of the vehicle.

9. The method of claim 1, wherein controlling one or more types of vehicle features includes communicating with an electronic control unit of the vehicle via peer to peer communication between the image capturing device and the electronic control unit of the vehicle and receiving settings data associated with the vehicle features associated with the one or more points of interest.

10. The method of claim 9, wherein the controlling one or more types of vehicle features includes receiving one or more user inputs on the plurality of vehicle feature settings sub-menu icons on the vehicle feature settings sub-menu associated with the corresponding vehicle feature and communicating with the electronic control unit of the vehicle to change settings data associated with the corresponding vehicle features.

11. A system for providing an augmented reality vehicle interface, comprising:

an image capturing device for capturing an image of the vehicle;

an augmented reality vehicle interface application that is executed on the image capturing device;

an image processing module that is included as a module of the augmented reality vehicle interface application that identifies the directional orientation and positional location of the image capturing device with respect to the vehicle;

an image recognition module that is included as a module of the augmented reality vehicle interface application that identifies points of interest that correspond with vehicle features within portions of the image of the vehicle captured by the image capturing device;

a user processing module that is included as a module of the augmented reality vehicle interface application that identifies a user classification category of the user that is capturing an image of the vehicle; and

a virtual reality module that is included as a module of the augmented reality vehicle interface application that overlays one or more virtual user interface objects on the points of interests.

12. The system of claim 11, wherein the image capturing device includes a portable electronic device with a camera, a control unit, memory, a touch screen display, a communication device, and a plurality of sensors.

13. The system of claim 11, wherein the image processing module determines the directional orientation and positional location of the image capturing device with respect to the vehicle to identify the points of interest by utilizing the plurality of sensors included within the portable electronic device and a digital compass included within the vehicle to thereby identify the direction and position from which the image is captured.

14. The system of claim 11, wherein the user processing module establishes a user classification category by determining if a driver or passenger is operating the portable electronic device when capturing the image of the vehicle by gathering data from the image processing module to locate the cardinal and ordinal position of the portable electronic device with respect to the vehicle upon the user capturing the image of the vehicle.

18. The system of claim 11, wherein the communication device of the portable electronic device communicates with an electronic control unit of the vehicle via peer to peer communication and receives settings data associated with the vehicle features associated with points of interest.

19. The system of claim 18, wherein the communication device of the portable electronic device communicates with the electronic control unit of the vehicle to change settings data associated with the vehicle features associated with the points of interest based on the user input received on the one or more virtual user interface objects of the augmented reality user interface presented on the touch screen display.

15. The system of claim 15, wherein overlaying one or more virtual user interface objects that correspond to the vehicle features on the points of interests includes the virtual reality module presenting an augmented reality user interface of the vehicle that includes one or more vehicle feature interface icons overlaid upon the points of interest associated with one or both of the driver or passenger that is operating the portable electronic device when capturing the image of the vehicle.

16. The system of claim 15, wherein overlaying one or more virtual user interface objects that correspond to the vehicle features on the points of interests includes the virtual reality module presenting an augmented reality user interface of the vehicle that includes a plurality of vehicle feature settings menu icons on a vehicle feature settings menu that are utilized to determine the settings associated with the vehicle features associated with one or both of the driver or passenger that is operating the portable electronic device when capturing the image of the vehicle.

17. The system of claim 16, wherein overlaying one or more virtual user interface objects that correspond to the vehicle features on the points of interests includes the virtual reality module presenting an augmented reality user interface of the vehicle that includes a plurality of vehicle feature settings sub-menu icons on a vehicle feature settings sub-menu that are utilized to determine the settings associated with the vehicle features associated with one or both of the driver or passenger that is operating the portable electronic device when capturing the image of the vehicle.

20. A computer readable medium comprising instructions that when executed by a processor perform actions comprising:

receiving an image of a vehicle with an image capturing device;

identifying points of interest within portions of the image of the vehicle that correspond with vehicle features;

identifying a user classification category of the user that is capturing an image of the vehicle;

communicating with an electronic control unit of the vehicle by computer communication between the image capturing device and the electronic control unit of the vehicle;

presenting an augmented reality image of the vehicle by overlaying one or more virtual user interface objects on the points of interest; and

controlling vehicle features via the one or more virtual user interface objects.