System and method for providing augmented reality based directions based on verbal and gestural cues

• US 10,339,711 B2
• Filed: 10/16/2014
• Issued: 07/02/2019
• Est. Priority Date: 03/15/2013
• Status: Expired due to Fees

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ng-Thow-Hing, Victor, Tran, Cuong, Bark, Karlin
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US14/515,620
• Publication Number:
  US 20150062168A1
• Time in Patent Office: 1,720 Days
• Field of Search: 345633
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A method and system for providing augmented reality based directions. The method and system include receiving a voice input based on verbal cues provided by one or more vehicle occupants in a vehicle. The method and system also include receiving a gesture input and a gaze input based on gestural cues and gaze cues provided by the one or more vehicle occupants in the vehicle. The method and system additionally include determining directives based on the voice input, the gesture input and the gaze input and associating the directives with the surrounding environment of the vehicle. Additionally, the method and system include generating augmented reality graphical elements based on the directives and the association of the directives with the surrounding environment of the vehicle. The method and system further include displaying the augmented reality graphical elements on a heads-up display system of the vehicle.

Claims

1. A method for providing augmented reality based directions, comprising:

receiving a voice input based on verbal cues provided by a passenger in a vehicle;

receiving a gesture input and a gaze input based on gestural cues and gaze cues provided by the passenger in the vehicle, wherein the gaze input of the passenger includes an object in a surrounding environment and an associated focal plane for the object, wherein the gaze input of the passenger is indicative of where the passenger is looking;

determining directives based on the voice input, the gesture input and the gaze input;

associating the directives with the surrounding environment of the vehicle;

generating augmented reality graphical elements based on the directives and the association of the directives with the surrounding environment of the vehicle; and

displaying, for a driver of the vehicle, the augmented reality graphical elements on a heads-up display system of the vehicle using an actuator to move a projector based on a location of the object in the surrounding environment to project the augmented reality graphical elements at the focal plane associated with the object, wherein the augmented reality graphical elements are displayed at the focal plane associated with the object from the gaze input of the passenger where the passenger is looking.

2. The method of claim 1, comprising generating visual directive data by correlating the gesture input and the gaze input.

3. The method of claim 2, wherein determining directives include correlating the voice input with the visual directive data with respect to objects in the surrounding environment of the vehicle.

4. The method of claim 3, wherein associating the directives with the surrounding environment of the vehicle comprises determining locational coordinates of the vehicle with respect to objects in the surrounding environment of the vehicle.

5. The method of claim 4, wherein associating the directives with the surrounding environment of the vehicle comprises determining characteristics of the objects in the surrounding environment of the vehicle.

6. The method of claim 5, wherein generating the augmented reality graphical elements comprises evaluating at least one of a plurality of rules based on at least one of the directives, the location coordinates of the vehicle with respect to objects in the surrounding environment of the vehicle, and the characteristics of the objects in the surrounding environment of the vehicle.

7. The method of claim 6, wherein displaying the augmented reality graphical elements on the heads-up display system of the vehicle comprises positioning the augmented reality graphical elements spatially and contextually with the surrounding environment of the vehicle as shown through the heads-up display.

8. A system for providing augmented reality based directions, comprising,

a voice recognition subsystem that provides a voice input based on verbal cues provided by a passenger in a vehicle;

a gesture recognition subsystem that provides a gesture input and a gaze input based on gestural cues and gaze cues provided by the passenger in the vehicle, wherein the gaze input of the passenger includes an object in a surrounding environment and an associated focal plane for the object, wherein the gaze input of the passenger is indicative of where the passenger is looking;

a command interpreter, implemented via a processor, that determines directives based on the voice input provided by the voice recognition subsystem, and the gesture input and the gaze input provided by the gesture recognition subsystem and associates the directives with the surrounding environment of the vehicle,

wherein the command interpreter generates augmented reality graphical elements based on the directives and the association of the directives with the surrounding environment of the vehicle, and

a heads up display system displays the augmented reality graphical elements generated by the command interpreter for a driver of the vehicle using an actuator to move a projector based on a location of the object in the surrounding environment to project the augmented reality graphical elements at the focal plane associated with the object,

wherein the augmented reality graphical elements are displayed, for the driver of the vehicle, at the focal plane associated with the object from the gaze input of the passenger where the passenger is looking.

9. The system of claim 8, comprising a gaze detection subsystem providing the gaze input to the gesture recognition subsystem, wherein the gesture recognition subsystem generates visual directive data by correlating the gesture input and the gaze input.

10. The system of claim 8, wherein the command interpreter determines directives by correlating the voice input provided by the voice recognition subsystem with the visual directive data provided by the gesture recognition subsystem with respect to objects in the surrounding environment of the vehicle.

11. The system of claim 8, comprising a GPS/Navigation subsystem that determines locational coordinates of the vehicle with respect to objects surrounding the environment of the vehicle.

12. The system of claim 8, comprising an environment sensing and modeling subsystem that determines characteristics of the objects in the surrounding environment of the vehicle.

13. The system of claim 12, wherein the command interpreter generates augmented reality graphical elements based on an evaluation of at least one of a plurality of rules based on at least one of the directives, the location coordinates of the vehicle with respect to objects in the surrounding environment of the vehicle provided by a GPS/Navigation subsystem, and the characteristics of the objects in the surrounding environment of the vehicle provided by the environment sensing and modeling subsystem.

14. The system of claim 13, wherein the heads up display system positions the augmented reality graphical elements spatially and contextually with the surrounding environment of the vehicle.

15. A non-transitory computer readable-storage medium storing instructions that when executed by a processor perform actions, comprising:

receiving a voice input from a passenger in a vehicle;

receiving a gesture input and a gaze input from the passenger in the vehicle, wherein the gaze input of the passenger includes an object in a surrounding environment and an associated focal plane for the object, wherein the gaze input of the passenger is indicative of where the passenger is looking;

determining directives based on the voice input, the gesture input and the gaze input;

associating the directives with the surrounding environment of the vehicle;

generating augmented reality graphical elements based on the directives and the association of the directives with the surrounding environment of the vehicle; and

displaying, for a driver of the vehicle, the augmented reality graphical elements on a heads-up display system of the vehicle using an actuator to move a projector based on a location of the object in the surrounding environment to project the augmented reality graphical elements at the focal plane associated with the object, wherein the augmented reality graphical elements are displayed at the focal plane associated with the object from the gaze input of the passenger where the passenger is looking.

16. The non-transitory computer readable-storage medium of claim 15, comprising generating visual directive data by correlating the gesture input and the gaze input.

17. The non-transitory computer readable-storage medium of claim 16, wherein associating the directives with the surrounding environment of the vehicle comprises determining locational coordinates of the vehicle with respect to objects in the surrounding environment of the vehicle.

18. The non-transitory computer readable-storage medium of claim 17, wherein associating the directives with the surrounding environment of the vehicle comprises determining characteristics of the objects in the surrounding environment of the vehicle.

19. The non-transitory computer readable-storage medium of claim 18, wherein generating the augmented reality graphical elements comprises evaluating at least one of a plurality of rules based on at least one of the directives, the location coordinates of the vehicle with respect to objects in the surrounding environment of the vehicle, and the characteristics of the objects in the surrounding environment of the vehicle.

20. The non-transitory computer readable-storage medium of claim 19, wherein displaying the augmented reality graphical elements on the heads-up display system of the vehicle comprises positioning the augmented reality graphical elements spatially and contextually with the surrounding environment of the vehicle as shown through the heads-up display.