Graphical vehicle command system for autonomous vehicles on full windshield head-up display

• US 9,008,904 B2
• Filed: 12/30/2010
• Issued: 04/14/2015
• Est. Priority Date: 12/30/2010
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Szczerba, Joseph F., Seder, Thomas A., Cui, Dehua
• Primary Examiner(s): TISSOT, ADAM D
• Application Number:
  US12/981,602
• Publication Number:
  US 20120173067A1
• Time in Patent Office: 1,566 Days
• Field of Search: 701/36, 701/300, 701/49, 345/7, 340/691.6
• US Class Current: 701/36
• CPC Class Codes: B60K 2360/141 Activation of instrument input devices by approaching fingers or pens
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 2360/785 on or in relation to the windshield or windows
• CPC Class Codes: B60K 2370/141 Activation by approaching fingers or pens
• CPC Class Codes: B60K 2370/334 Projection means
• CPC Class Codes: B60K 2370/785 on or in relation to the windshield or windows
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact

Abstract

A method to control operation of a vehicle based upon a user input from a user of the vehicle to a graphic projection display representing a registered graphic to a view in front of the vehicle includes projecting the registered graphic to the view in front of the vehicle upon the graphic projection display, monitoring the user input to the graphic projection display, and controlling the operation of the vehicle based upon the user input to the graphic projection display.

Claims

1. Method to control operation of a vehicle based upon a user input from a user of the vehicle to a graphic projection display representing a registered graphic to a view in front of the vehicle, the method comprising:

monitoring, at an enhanced vision system (EVS) system manager, a user eye location;

projecting, at a graphics system, the registered graphic to the view in front of the vehicle at a location upon the graphic projection display, comprising;

monitoring, at the EVS system manager, a command selection to be presented to the user of the vehicle,

monitoring, at the EVS system manager, the view in front of the vehicle,

monitoring, at the EVS system manager, a location of an object or feature corresponding to the command selection using at least one of vehicle sensors and a wireless connection, the object or feature visible through a transparent windscreen of the vehicle in the view in front of the vehicle;

determining, at EVS system manager, the registered graphic representing the command selection based upon the view in front of the vehicle, and the user eye location, wherein the location of the registered graphic projected upon the graphic projection display is indicated to the location of the object or feature visible through the transparent windscreen of the vehicle;

monitoring, at the EVS system manager, an estimated user gaze location based on the monitored user eye location, the user gaze location corresponding to a point on the view in front of the vehicle where eyes of the user are currently focused;

monitoring, at the EVS system manger, the user input to the graphic projection display, the user input comprising a hand gesture of the user,

determining, at the EVS system manager, a portion of the graphic projection display at which the user input is indicated based upon the estimated user gaze location; and

activating, at the EVS system manager, the command selection based on the portion of the graphic projection display at which the user input is indicated to include the location of the registered graphic upon the graphic projection display.

2. The method of claim 1, wherein the graphic projection display is a substantially full-windscreen transparent windscreen head -up display.

3. The method of claim 1, wherein the graphic projection display is a transparent windscreen head-up display.

4. The method of claim 1, wherein monitoring the command selection comprises monitoring a selection to be presented regarding a navigational route of the vehicle.

5. The method of claim 4, wherein monitoring the selection to be presented regarding the navigational route of the vehicle comprises:

monitoring traffic along a planned route of the vehicle; and

determining the selection be presented comprising an alternative route based upon the monitored traffic.

6. The method of claim 1, wherein monitoring the command selection comprises:

monitoring a user input indicated to the object or feature in the view in front of the vehicle;

retrieving information regarding the object or feature through the wireless connection; and

determining the command selection to be presented based upon the user input indicated to the object or feature and the retrieved information.

7. The method of claim 1, further comprising operating the vehicle through one of an autonomous method to operate the vehicle and a semi-autonomous method to operate the vehicle;

and

wherein monitoring the command selection comprises;

monitoring information regarding operating the vehicle through the one of the autonomous method to operate the vehicle and the semi-autonomous method to operate the vehicle;

determining configurable options for operating the vehicle through the one of the autonomous method to operate the vehicle and the semi-autonomous method to operate the vehicle based upon the monitored information; and

determining the command selection to be presented to the user of the vehicle based upon determined configurable options.

8. The method of claim 7, wherein operating the vehicle through the one of the autonomous method to operate the vehicle and the semi-autonomous method to operate the vehicle comprises operating the vehicle through one of an adaptive cruise control method, a lane keeping method, a lane change method, a collision preparation method, and a collision avoidance method.

9. The method of claim 3, wherein monitoring the user input to the graphic projection display comprises monitoring a hand gesture indicated to the graphic projection display.

10. The method of claim 9, wherein monitoring the hand gesture comprises utilizing an imaging device to generate three-dimensional mapping of the hand gesture;

and

determining the hand gesture to indicate to the registered graphic based upon the three- dimensional mapping of the hand gesture.

11. The method of claim 10, wherein utilizing the imaging device to generate three-dimensional mapping of the hand gesture comprises capturing images with a device utilizing one of infrared light, ultraviolet light, and a combination of infrared and ultraviolet light.

12. The method of claim 10, wherein utilizing the imaging device to generate three-dimensional mapping of the hand gesture comprises utilizing a laser speckle pattern.

13. The method of claim 10, wherein utilizing the imaging device to generate three-dimensional mapping of the hand gesture comprises utilizing pulsed modulation.

14. The method of claim 10, wherein utilizing the imaging device to generate three-dimensional mapping of the hand gesture comprises illuminating the hand gesture with beams of radiation and generating the three-dimensional mapping based upon intensity-based mapping.

15. The method of claim 1, wherein the graphic projection display comprises a touch screen device representing the view in front of the vehicle and the location of the registered graphic upon the touch screen device indicated to the location of the object or feature in the view.

16. The method of claim 1, wherein the graphic projection display comprises a human machine interface device representing the view in front of the vehicle and the location of the registered graphic upon the human machine interface indicated to the location of the object or feature in the view.

17. The method of claim 16, wherein the human machine interface device comprises a non-transparent surface comprising luminescent materials;

wherein projecting the registered graphic to the view in front of the vehicle upon the graphic projection display comprises;

projecting the view upon the graphic projection display;

determining the registered graphic based upon the projected view; and

projecting the registered graphic upon the graphic projection display.

18. The method of claim 1, wherein projecting the registered graphic comprises projecting the registered graphic upon a pixelated field of view limited architecture.

19. Apparatus to control operation of a vehicle based upon a user input from a user of the vehicle to a graphic projection display representing a registered graphic to a view in front of the vehicle, the apparatus comprising:

the graphic projection display; and

an enhanced vision system (EVS) system manager;

monitoring a user eye location;

monitoring an estimated user gaze location based on the user eye location, the user gaze location corresponding to a point on the view in front of the vehicle where eyes of the user are currently focused

determining the registered graphic to be represented based upon the view in front of the vehicle, the registered graphic representing a command selection to be presented to the user of the vehicle, wherein determining the registered graphic comprises;

monitoring a location of an object or feature corresponding to the command selection using at least one of vehicle sensors and a wireless connection, the object or feature visible through a transparent windscreen of the vehicle in the view in front of the vehicle;

determining the registered graphic representing the command selection based upon the view in front of the vehicle and the user eye location, wherein the registered graphic is projected at a location upon the graphic projection display indicated to the location of the object or feature visible through the transparent windscreen of the vehicle;

monitoring the user input to the graphic projection display, the user input comprising a hand gesture of the user;

determining a portion of the graphic projection display at which the user input is indicated based upon the estimated user gaze location; and

activating the command selection based on the portion of the graphic projection display at which the user input is indicated to include the location of the registered graphic upon the graphic projection display.