Systems and methods for a parallel autonomy interface

• US 10,311,726 B2
• Filed: 07/21/2017
• Issued: 06/04/2019
• Est. Priority Date: 07/21/2017
• Status: Expired due to Fees

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Patrikalakis, Andrew, Michel, Philipp
• Primary Examiner(s): Foxx, Chico A
• Application Number:
  US15/656,422
• Publication Number:
  US 20190027033A1
• Time in Patent Office: 683 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/178 Warnings
• CPC Class Codes: B60K 2360/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 2360/191 Highlight information
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G08G 1/09626 where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G09G 2380/10 Automotive applications

Abstract

A parallel autonomy interface system can include processing circuitry communicably coupling one or more sensors of an autonomous vehicle and one or more displays of the autonomous vehicle. The processing circuitry can be configured to receive output from the one or more sensors, determine if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors, activate an alert in response to detecting the potential danger, the alert corresponding to avatar motion, and enable the avatar to indicate a direction of the potential danger in response to activation of the alert.

Claims

1. A parallel autonomy interface system, comprising:

one or more sensors coupled to a vehicle capable of autonomous operation;

one or more displays configured to display an avatar, the avatar including at least a head and a hand; and

processing circuitry communicably coupling the one or more sensors and the one or more displays, the processing circuitry configured to

receive output from the one or more sensors,

determine if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors,

activate an alert in response to detecting the potential danger, the alert corresponding to a change in a size of the avatar, and

cause the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing hand of the avatar towards the potential danger.

2. The parallel autonomy interface system of claim 1, wherein the processing circuitry is configured to

display the avatar in a first location, the first location being within a predetermined area relative to a first focal point of the operator of the vehicle,

determine if the first focal point changes to a second focal point of the operator of the vehicle,

determine if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point, and

display the avatar in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

3. The parallel autonomy interface system of claim 2, wherein the predetermined area corresponds to a predetermined range of peripheral vision of the operator of the vehicle.

4. The parallel autonomy interface system of claim 1, wherein the processing circuitry is configured to

determine if the one or more displays are producing sound, and

reduce a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.

5. The parallel autonomy interface system of claim 2, wherein the first location corresponds to a first display and the second location corresponds to a second display.

6. The parallel autonomy interface system of claim 1, wherein the avatar can be selected by the operator from a plurality of avatars.

7. The parallel autonomy interface system of claim 1, wherein the avatar further indicates the direction of the potential danger by walking in the direction of the potential danger.

8. A method of operating a parallel autonomy interface, comprising:

receiving output from one or more sensors, the one or more sensors communicably coupled to one or more displays;

determining, via processing circuitry, if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors;

activating an alert in response to detecting the potential danger, the alert corresponding to a change in a size of a displayed avatar, the avatar including at least a head and a hand; and

causing the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing the hand of the avatar towards the potential danger.

9. The method of claim 8, further comprising:

displaying the avatar, via the one or more displays, in a first location, the first location being within a predetermined area relative to a first focal point of an operator of the vehicle;

determining if the first focal point changes to a second focal point of the operator of the vehicle;

determining if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point; and

displaying the avatar, via the one or more displays, in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

10. The method of claim 9, wherein the predetermined area corresponds to a predetermined range of peripheral vision of the operator of the vehicle.

11. The method of claim 8, further comprising:

determining if the one or more displays are producing sound; and

reducing a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.

12. The method of claim 9, wherein the first location corresponds to a first display and the second location corresponds to a second display.

13. The method of claim 8, wherein the avatar can be selected by an operator of the vehicle from a plurality of avatars.

14. The method of claim 8, wherein the avatar further indicates the direction of the potential danger by walking in the direction of the potential danger.

15. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method of operating a parallel autonomy interface, comprising:

receiving output from one or more sensors, the one or more sensors communicably coupled to one or more displays;

determining if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors;

activating an alert in response to detecting the potential danger, the alert corresponding to a change in a size of a displayed avatar, the avatar including at least a head and a hand; and

causing the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing the hand of the avatar towards the potential danger.

16. The method of claim 15, further comprising:

displaying the avatar, via the one or more displays, in a first location, the first location being within a predetermined area relative to a first focal point of an operator of the vehicle;

determining if the first focal point changes to a second focal point of the operator of the vehicle;

determining if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point; and

displaying the avatar, via the one or more displays, in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

17. The method of claim 15, further comprising:

determining if the one or more displays are producing sound; and

reducing a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.