Control of display device for autonomous vehicle

• US 10,303,045 B1
• Filed: 12/20/2017
• Issued: 05/28/2019
• Est. Priority Date: 12/20/2017
• Status: Active Grant

• Current Assignee: Micron Technology, Inc.
• Original Assignee: Micron Technology, Inc.
• Inventors: Sato, Junichi
• Primary Examiner(s): Chowdhury, Sultan
• Assistant Examiner(s): Cruz, Magda
• Application Number:
  US15/848,630
• Publication Number:
  US 20190187809A1
• Time in Patent Office: 524 Days
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G03B 21/006 using LCD's
• CPC Class Codes: G03B 21/147 Optical correction of image distortions, e.g. keystone
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 3/08 Projecting images onto non-planar surfaces, e.g. geodetic screens
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A display device of an autonomous vehicle is controlled based on data collected from sensors located in or on the vehicle. The display device is used to present one or more images to a driver and/or passengers of the autonomous vehicle. The display device can be, for example, a windshield and/or other window of the vehicle. Image data can be, for example, transformed to improve visual perception by passengers in the vehicle when the images are displayed on a curved shape of the windshield.

Claims

1. A method, comprising:

collecting, by at least one processor, data from at least one sensor in an autonomous vehicle;

analyzing, by the at least one processor, the collected data from the at least one sensor, the analyzing comprising providing the collected data as an input to a computer model;

transforming image data to correct for distortion associated with a shape of a windshield, wherein the transformed image data corresponds to at least one first image; and

controlling, based on the analyzing the collected data, a display device of the autonomous vehicle, wherein the controlling comprises performing an action based on an output from the computer model, wherein the display device comprises the windshield, and wherein controlling the display device comprises presenting, based on the transformed image data, the at least one first image on the windshield.

2. The method of claim 1, wherein the at least one sensor comprises a sensor of a wearable computing device worn by a passenger of the autonomous vehicle.

3. The method of claim 1, further comprising training the computer model using training data including at least a portion of the collected data.

4. The method of claim 1, wherein collecting the data from the at least one sensor comprises receiving image data or audio data from the at least one sensor.

5. The method of claim 1, wherein the collected data comprises image data, and analyzing the collected data comprises performing facial recognition on the image data to identify facial features of at least one passenger of the autonomous vehicle.

6. The method of claim 5, wherein performing the facial recognition comprises extracting features from an image of a face of a first passenger to determine an emotional state of the first passenger.

7. The method of claim 1, wherein the collected data comprises biometric data corresponding to at least one passenger located in the autonomous vehicle.

8. The method of claim 1, wherein the at least one sensor comprises at least one of a motion detector, a camera, an accelerometer, or a microphone.

9. The method of claim 1, wherein the display device comprises at least one window of the autonomous vehicle, wherein the at least one window is transparent to permit passenger viewing of an environment outside of the autonomous vehicle, and wherein controlling the display device comprises changing a state of the at least one window to an opaque state such that the passenger viewing is blocked.

10. The method of claim 9, wherein controlling the display device further comprises, subsequent to changing the state of the at least one window to the opaque state, generating an image on the at least one window, wherein the image is for viewing by a passenger of the autonomous vehicle.

11. The method of claim 1, wherein the display device comprises a liquid crystal display integrated into the windshield.

12. A system for an autonomous vehicle, the system comprising:

at least one sensor;

a display device;

at least one processor; and

memory storing instructions configured to instruct the at least one processor to;

collect data from the at least one sensor;

analyze the collected data, wherein the analyzing comprises providing the data as an input to a machine learning model;

transform image data to correct for distortion associated with a shape of a windshield, wherein the transformed image data corresponds to at least one image; and

control, based on the analyzing the collected data, the display device, wherein the controlling comprises performing an action based on at least one output from the machine learning model, wherein the display device comprises the windshield, and wherein controlling the display device comprises presenting, based on the transformed image data, the at least one image on the windshield.

13. The system of claim 12, wherein the display device comprises a liquid crystal display, and performing the action comprises generating the at least one image for display by the liquid crystal display for viewing by a passenger of the autonomous vehicle.

14. The system of claim 12, further comprising a projector mounted in the autonomous vehicle, wherein controlling the display device comprises projecting the at least one image onto a surface of the windshield.

15. The system of claim 12, wherein the display device comprises at least one window of the autonomous vehicle, and controlling the display device comprises changing a state of the at least one window to a transparent state that permits passenger viewing outside of the autonomous vehicle, wherein the instructions are further configured to instruct the at least one processor to:

based on the at least one output from the machine learning model, select a route for controlling navigation of the autonomous vehicle.

16. The system of claim 12, further comprising a communication interface configured to:

wirelessly transmit the collected data to a computing device; and

receive training data from the computing device;

wherein a configuration of the machine learning model is updated using the training data.

17. The system of claim 16, wherein the training data comprises sensor data collected from at least one other autonomous vehicle.

18. The system of claim 16, wherein the training data comprises at least one of:

data collected by sensors that monitor at least one physical activity of persons inside fixed structures;

electronic communications of persons;

data regarding times of day and corresponding actions performed by the autonomous vehicle while carrying prior passengers;

biometric data for prior passengers that have traveled in the autonomous vehicle;

or

data regarding input selections made in a user interface of the autonomous vehicle by prior passengers.

19. A non-transitory computer storage medium storing instructions which, when executed on a computing device, cause the computing device to perform a method for an autonomous vehicle, the method comprising:

determining that the autonomous vehicle is in an automatic navigation mode;

in response to determining that the autonomous vehicle is in the automatic navigation mode, controlling a windshield of the autonomous vehicle to provide a display for viewing by at least one passenger;

transforming image data to correct for distortion associated with a shape of the windshield, wherein the transformed image data corresponds to at least one image; and

presenting, based on the transformed image data, the at least one image on the windshield as part of providing the display for viewing by the at least one passenger.

20. The non-transitory computer storage medium of claim 19, wherein distortion correction provided by transforming the image data flattens a visual presentation of the at least one image for viewing by a first passenger of the at least one passenger.