Vision augmentation for supplementing a person's view

• US 10,332,292 B1
• Filed: 01/17/2017
• Issued: 06/25/2019
• Est. Priority Date: 01/17/2017
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Arnicar, Taylor Andrew
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US15/407,788
• Time in Patent Office: 889 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/178 Warnings
• CPC Class Codes: B60K 2360/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 2360/31 Virtual images
• CPC Class Codes: B60K 2370/1529 Head-up displays
• CPC Class Codes: B60K 2370/177 Augmented reality
• CPC Class Codes: B60K 2370/178 Warnings
• CPC Class Codes: B60K 2370/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 2370/193 for improving awareness
• CPC Class Codes: B60K 2370/31 Virtual images
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 37/06 of controls, e.g. controls knobs
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/00272 relying on extrapolation of current movement
• CPC Class Codes: G01C 21/365 : Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01C 21/3863 : Structures of map data
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An augmentation system may supplement a person'"'"'s view from at least one of an actual vehicle or a virtual vehicle of an environment through which the vehicle travels. The system may include a receiver configured to receive data signals representative of operation of the vehicle and representative of objects in the environment through which the vehicle travels. The system may also include a converter in communication with the receiver. The converter may be configured to convert the data signals into display data representative of the operation of the vehicle and representative of the objects in the environment through which the vehicle travels. The converter may also be configured to communicate the display data to a display in communication with the converter and configured to display, within the person'"'"'s view from the vehicle, images representative of the display data to supplement the person'"'"'s view.

Claims

1. An augmentation system for supplementing a person'"'"'s view from a vehicle of an environment through which the vehicle travels, the augmentation system comprising:

a sensor on the vehicle configured to generate sensor data, the sensor data representative of an object in the environment through which the vehicle travels;

a display; and

one or more processors configured to;

determine a direction of view of the person;

determine, based at least in part on the direction of view, a subset of the sensor data;

determine, based at least in part on the subset of the sensor data, a representation of the object in the environment through which the vehicle travels; and

cause the display to augment the person'"'"'s view of the environment through which the vehicle travels by presenting the subset of the sensor data superimposed with the representation of the object,

wherein the representation of the object is indicative of at least one of a location of the object, one or more extents of the object, a type of the object, a historical path of the object, or a predicted path of the object.

2. The augmentation system of claim 1, wherein the one or more processors are further configured to:

determine vehicle data signals representative of operation of the vehicle,

wherein the vehicle data signals representative of operation of the vehicle comprise at least one of signals associated with a trajectory of the vehicle or signals associated with a predicted trajectory of the vehicle, and wherein the vehicle data signals are further configured to display images representative of at least one of the trajectory of the vehicle or the predicted trajectory of the vehicle.

3. The augmentation system of claim 1, wherein the representation of the object in the environment through which the vehicle travels comprises one or more bounding boxes.

4. The augmentation system of claim 1, wherein the display comprises at least one of a video headset configured to be worn by the person, a heads-up display configured to be associated with the vehicle, or video glasses.

5. The augmentation system of claim 1, further comprising at least one of a viewer position or orientation system configured to estimate at least one of a position of the person or the direction of view of the person, with respect to the vehicle.

6. The augmentation system of claim 1, wherein at least one of the images comprises an image representative of a stationary object and providing an indication that the stationary object is stationary.

7. The augmentation system of claim 6, wherein the stationary object is at least one of an edge of a road on which the vehicle travels, a sign, a traffic signal, or an object on the road, and the image provides an indication of at least one of the location or a type of object.

8. The augmentation system of claim 1, wherein at least one of the images comprises an image representative of a moving object and providing an indication that the moving object is moving.

9. The augmentation system of claim 8, wherein the moving object is at least one of a vehicle, a pedestrian, a cyclist, or an animal, and the image provides an indication of at least one of a location of the object, a type of the object, a historical path of the object, or a predicted path of the object.

10. The augmentation system of claim 1, wherein at least one of the images comprises an image providing an indication of a predicted collision with the object, and wherein the indication of a predicted collision comprises at least one of an addition of a textual overlay to the image, a change of color of the image, a change of shade of color of the image, a change of shape of the image, a change in size of the image, blinking of the image, or a change in rate of blinking of the image.

11. The augmentation system of claim 10, wherein the indication of a predicted collision changes based at least in part on a change in probability of the predicted collision occurring.

12. The augmentation system of claim 1, wherein the display, when augmenting the person'"'"'s view, is configured to cause the person to control the vehicle based at least in part on the comparison.

13. The augmentation system of claim 1, wherein at least one of the images provides an indication of a vehicle operation law or regulation.

14. The augmentation system of claim 13, wherein the indication of a vehicle operation law or regulation comprises an indication of at least one of a requirement to stop the vehicle, a maximum speed of the vehicle, or a requirement to yield right-of-way to at least one of a pedestrian, a cyclist, or another vehicle.

15. The augmentation system of claim 1, wherein the sensor comprises one or more of a LIDAR sensor, a camera, or RADAR.

16. A method for supplementing a person'"'"'s view from at least one of an actual vehicle or a virtual vehicle of an environment through which the vehicle travels, the method comprising:

receiving data signals representative of operation of the vehicle and representative of objects in the environment through which the vehicle travels;

converting the data signals into display data representative of the operation of the vehicle and representative of the objects in the environment through which the vehicle travels;

displaying images representative of the operation of the vehicle and images representative of the objects in the environment through which the vehicle travels to a person to supplement the person'"'"'s view, such that the images representative of the operation of the vehicle and the images representative of the objects in the environment through which the vehicle travels are superimposed onto a representation of the environment to augment the person'"'"'s view of the environment;

comparing, as a comparison, at least one of objects or vehicle motion displayed in the representation of the environment with the images representative of the operation of the vehicle and the images representative of the objects in the environment through which the vehicle travels;

identifying, based at least in part on the comparison, differences between the objects displayed in the representation of the environment and the images representative of the operation of the vehicle and the images representative of the objects in the environment through which the vehicle travels; and

controlling the vehicle based at least in part on the differences,

wherein the images representative of the objects in the environment through which the vehicle travels comprise data associated with at least one of a location of the objects, types of the objects, historical paths of the objects, or predicted paths of the objects.

17. The method of claim 16, wherein receiving the data signals representative of operation of the vehicle comprises receiving at least one of signals associated with a trajectory of the vehicle or signals associated with a predicted trajectory of the vehicle, and wherein displaying images representative of the operation of the vehicle comprises displaying images representative of at least one of the trajectory of the vehicle or the predicted trajectory of the vehicle.

18. The method of claim 16, wherein displaying the images representative of the operation of the vehicle, the images representative of the objects in the environment through which the vehicle travels, and the representation of the environment occurs substantially simultaneously.

19. The method of claim 16, wherein displaying images representative of the operation of the vehicle and the images representative of the objects in the environment through which the vehicle travels comprises displaying the images via at least one of a video headset configured to be worn by a person, a heads-up display configured to be associated with a windshield of the vehicle, or video glasses.

20. The method of claim 16, further comprising estimating at least one of a position of the person or a direction of view of the person and displaying the images based in part on the at least one of the position or the direction of view.

21. The method of claim 16, wherein at least one of the images comprises an image representative of a stationary object, and the method comprises displaying an indication that the stationary object is stationary.

22. The method of claim 21, wherein the stationary object is at least one of an edge of a road on which the vehicle travels, a sign, a traffic signal, or an object on the road, and the method comprises displaying at least one of the location or the type of the stationary object.

23. The method of claim 16, wherein at least one of the images comprises an image representative of a moving object, and the method comprises displaying an indication that the moving object is moving.

24. The method of claim 23, wherein the moving object is at least one of a vehicle, a pedestrian, a cyclist, or an animal, and the method comprises displaying at least one of a location of the moving object, a type of the moving object, a historical path of the moving object, or a predicted path of the moving object.

25. The method of claim 16, wherein displaying the images comprises displaying an image providing an indication of a predicted collision between the vehicle and at least one of the objects.

26. The method of claim 25, wherein displaying the indication of the predicted collision comprises at least one of adding a textual overlay to the image, changing a color of the image, changing a shade of color of the image, changing a shape of the image, changing a size of the image, causing the image to blink, or causing a change in a rate of blinking of the image.

27. The method of claim 26, wherein displaying the image providing the indication of the predicted collision comprises changing the image based at least in part on a change in probability of the predicted collision occurring.

28. The method of claim 16, wherein displaying the images comprises displaying an indication of a vehicle operation law or regulation.

29. The method of claim 28, wherein displaying the indication of the vehicle operation law or regulation comprises displaying an indication of at least one of a requirement to stop the vehicle, a maximum speed of the vehicle, or a requirement to yield right-of-way to at least one of a pedestrian, a cyclist, or another vehicle.

30. A method for testing an autonomous vehicle, the method comprising:

supplementing a person'"'"'s view from the autonomous vehicle of an environment through which the autonomous vehicle travels, the autonomous vehicle comprising an actual vehicle, the supplementing comprising;

receiving data signals representative of operation of the autonomous vehicle and representative of objects in the environment through which the autonomous vehicle travels;

converting the data signals into images representative of the operation of the autonomous vehicle and images representative of the objects in the environment through which the autonomous vehicle travels;

superimposing the images representative of the operation of the autonomous vehicle and the images representative of the objects in the environment through which the autonomous vehicle travels with the person'"'"'s view from the autonomous vehicle;

comparing, as a comparison, actual objects viewable by the person from the autonomous vehicle with the images representative of the operation of the autonomous vehicle and the images representative of the objects in the environment through which the autonomous vehicle travels; and

identifying, based at least in part on the comparison, differences between the actual objects and the images representative of the objects in the environment through which the autonomous vehicle travels.

31. The method of claim 30, further comprising collecting data associated with the differences.

32. The method of claim 30, further comprising analyzing the differences and modifying operation of at least one of sensors associated with the autonomous vehicle or control systems associated with the autonomous vehicle.

33. The method of claim 30, further comprising receiving at least one manual input to control the autonomous vehicle manually based in part on the differences.