Device, system, and method for pictorial language for autonomous vehicle

• US 10,011,285 B2
• Filed: 05/23/2016
• Issued: 07/03/2018
• Est. Priority Date: 05/23/2016
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: McNew, John-Michael
• Primary Examiner(s): Nguyen, Hung T
• Application Number:
  US15/161,608
• Publication Number:
  US 20170334459A1
• Time in Patent Office: 771 Days
• Field of Search: 340435, 340438-439, 340901, 701 1, 701 45, 345633
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/16 Type of output information
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/20 Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60R 11/0229 for displays, e.g. cathodic tubes
• CPC Class Codes: B60R 2300/108 using 'non-standard' camera systems, e.g. camera sensor used for additional purposes i.a. rain sensor, camera sensor split in multiple image areas
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/0088 Adaptive recalibration
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention

Abstract

The autonomous vehicle visual language system displays a plan sentence to an operator of an autonomous vehicle. The plan sentence includes visual syntax, the visual syntax being images displayed in predetermined configurations to convey specific information associated with any maneuvers the autonomous vehicle plans to execute. The visual syntax allows for a structured layering of information such that the plan sentence can display more complex information. The operator of the autonomous vehicle can process individual warnings and information more quickly, even when the operator has never seen a particular warning or element of information before. Additionally, the system 100 allows for personalization of a driver model to more closely reflect the operator'"'"'s driving style, which is then implemented in the visual language displayed as the plan sentence.

Claims

1. An autonomous vehicle pictorial language system comprising:

a plurality of sensors;

a driver preferences database;

a human-machine interface; and

a processing circuitry configured to

receive output from the plurality of sensors,

receive a driver model from the driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences, and

display a pictorial language via the human-machine interface in response to the output from the plurality of sensors and the driver model,

wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

2. The autonomous vehicle pictorial language system of claim 1, wherein the processing circuitry is further configured to

display the plan sentence,

receive a driver response sentence,

determine if the driver sentence received does not match the plan sentence more than a predetermined number of times,

change the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times,

change the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times, and

update the driver model in response to one or more changes to the first adverb and the noun.

3. The autonomous vehicle pictorial language system of claim 2, wherein driver confirmation of the plan sentence causes the vehicle to execute a maneuver displayed by the plan sentence when the driver would otherwise execute the maneuver manually.

4. The autonomous vehicle pictorial language system of claim 2, wherein the driver response sentence is constructed via output received from manual operations executed by the driver.

5. The autonomous vehicle pictorial language system of claim 2, wherein the pictorial language is displayed as the plan sentence.

6. The autonomous vehicle pictorial language system of claim 5, wherein a determination of the noun via the plurality of sensors and the driver model causes the human-machine interface to display a color associated with the noun based on the determination of the noun to construct the plan sentence.

7. The autonomous vehicle pictorial language system of claim 5, wherein a determination of the first adverb via the driver model, the verb, laws associated with a location of the vehicle, and vehicle rules causes the human-machine interface to display an image associated with the first adverb based on the determination of the first adverb to construct the plan sentence.

8. The autonomous vehicle pictorial language system of claim 5, wherein a determination of the object classifier via the plurality of sensors causes the human-machine interface to display an image associated with the object classifier based on the determination of the object classifier to construct the plan sentence.

9. The autonomous vehicle pictorial language system of claim 5, wherein a determination of the second adverb via the plurality of sensors and the object classifier causes the human-machine interface to display an image associated with the second adverb based on the determination to construct the plan sentence.

10. The autonomous vehicle pictorial language system of claim 5, wherein the pictorial language includes visual syntax such that visual elements share a common basis that allow for structured layering of information to display complex information.

11. The autonomous vehicle pictorial language system of claim 6, wherein the plurality of sensors further determines road structure and sensor quality, such that the sensor quality further determines system health, to determine the noun.

12. The autonomous vehicle pictorial language system of claim 1, wherein the plurality of sensors includes LIDAR, Radar, V2X, at least one camera, ultrasonics, and GPS.

13. The autonomous vehicle pictorial language system of claim 12, wherein a determination of the verb via the at least one camera, the Radar, and the GPS causes the human-machine interface to display an image associated with the verb based on the determination of the verb to construct the plan sentence.

14. The autonomous vehicle pictorial language system of claim 13, wherein the at least one camera, Radar, and GPS determine road structure, the GPS also being utilized to determine laws associated with a location of the vehicle, to determine the verb.

15. The autonomous vehicle pictorial language system of claim 14, wherein the at least one camera is used independently or in combination with the GPS to determine laws associated with the location of the vehicle.

16. A method for displaying a pictorial language executed by a processing circuitry and comprising:

receiving output from a plurality of sensors;

receiving a driver model from a driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences; and

displaying a pictorial language via a human-machine interface in response to the output from the plurality of sensors and the driver model,

wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

17. The method of claim 16, further comprising:

displaying the plan sentence;

receiving a driver response sentence;

determining, via the processing circuitry, if the driver sentence received does not match the plan sentence more than a predetermined number of times;

changing the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times;

changing the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times; and

updating the driver model in response to one or more changes to the first adverb and the noun.

18. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method comprising:

receiving output from a plurality of sensors;

receiving a driver model from a driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences; and

displaying a pictorial language via a human-machine interface in response to the output from the plurality of sensors and the driver model,

wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

19. The method of claim 18, further comprising:

displaying the plan sentence;

receiving a driver response sentence;

determining if the driver sentence received does not match the plan sentence more than a predetermined number of times;

changing the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times;

changing the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times; and

updating the driver model in response to one or more changes to the first adverb and the noun.