Intuitive occluded object indicator

• US 10,552,690 B2
• Filed: 11/04/2016
• Issued: 02/04/2020
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Schubert, Martin Friedrich, Watson, Philip Edwin
• Primary Examiner(s): Rush, Eric
• Application Number:
  US15/343,483
• Publication Number:
  US 20180129888A1
• Time in Patent Office: 1,187 Days
• Field of Search: 382100, 382103, 382104, 382107, 382291, 382302, 382312, 382325, 345 7, 345 9, 345418, 345419, 345421, 345619, 345629, 345632-634, 348 61, 348143, 348148, 3404255, 340435, 340436, 340944, 359613, 359618, 359619, 359629, 359630, 701301
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/202 displaying a blind spot scene on the vehicle part responsible for the blind spot
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Methods, systems, and apparatus for providing indications of occluded objects. In some aspects a method includes the actions of determining a position of an observer whose view of an object is obstructed by a barrier; determining a position of the object relative to the observer; generating an indication of the position of the object based at least in part on the determined position of the observer; and displaying the indication of the position of the object on a display located between the observer and the barrier.

Claims

1. A computer-implemented method comprising:

determining respective positions of multiple observers located in different vehicles, wherein the multiple observers have respective views of an area obstructed by a barrier;

selecting, for each of the multiple observers, a proper subset of multiple objects in the area, the proper subsets each respectively being selected based on (i) a speed and direction of movement of the observer to which the proper subset corresponds and (ii) speeds and directions of movement of the multiple objects in the area, the proper subsets each including a particular object of the multiple objects;

determining, for each of the multiple observers, a position of the particular object relative to the observer;

generating, for each of the multiple observers, an indication of the position of the particular object based at least in part on the determined position of the observer; and

concurrently displaying, on a display that is located outside the vehicles and that is located between the multiple observers and the barrier, the generated indications of the position of the particular object to the respective observers such that different indications of the position of the particular object are displayed to the respective observers based on the respective positions of the multiple observers and such that the display omits, for each of the multiple observers, indications of objects in the area that are not in the proper subset selected for the observer.

2. The method of claim 1, wherein concurrently displaying the generated indications of the position of the particular object comprises concurrently displaying, on a single display surface, different views of the particular object to each of the multiple observers based on the determined respective positions of the multiple observers.

3. The method of claim 2, wherein each different view of the particular object corresponds to a intersection of (i) a three-dimensional space between the respective observer'"'"'s field of vision and the particular object, and (ii) the display.

4. The method of claim 2, wherein each different view of the particular object comprises a representation of the particular object that has a size and position in a respective observer'"'"'s field of view that corresponds to a size and position that the particular object would have in the respective observer'"'"'s field of view if the barrier were not present.

5. The method of claim 1, further comprising dynamically changing the indications of the position of the particular object as the positions of the multiple observers and the particular object change.

6. The method of claim 1, further comprising:

detecting a set of objects behind the barrier at multiple times; and

filtering, from the set of detected objects, objects that have not moved more than a predetermined minimum amount over a period of time.

7. The method of claim 1, further comprising:

generating, for each of the multiple objects, a relevance score indicative of a predicted relevance of the object to a particular observer whose view of the object is obstructed by the barrier; and

selecting, for the particular observer, a proper subset of the multiple objects based on the generated relevance scores.

8. The method of claim 7, comprising displaying, on the display and to the particular observer, an indication of each of the objects in the proper subset of the multiple objects that is selected for the particular observer, without displaying objects of the multiple objects that are not in the proper subset selected for the particular observer.

9. The method of claim 7, wherein generating the relevance scores comprises generating, for each object of the multiple objects, a relevance score based on one or more of (i) a size of the object, (ii) a shape of the object, (iii) movement of the object, (iv) a trajectory of the object, (v) proximity of the object to the barrier, (vi) an image recognition score for the object, (vii) a position of the particular observer, (viii) a trajectory of the particular observer, (ix) a measure of the particular observer'"'"'s ability to brake or change trajectory, (x) historical data indicating objects in proximity to the barrier, (xi) historical data indicating similar types of barriers, or (xii) capabilities of the object.

10. The method of claim 7, further comprising, selecting a representation for a specific object of the multiple objects, from among multiple different representations for the specific object, based on the relevance score for the specific object.

11. The method of claim 1, further comprising augmenting, for at least one of the multiple observers, the generated indication of the position of the particular object based at least in part on the determined position of the observer with context or emphasis information.

12. The method of claim 1, wherein the display is stationary,

wherein at least a first observer of the multiple observers is in a moving vehicle such that the position of the moving vehicle changes relative to the display and the particular object, and

wherein the method comprises altering the indication of the position of the particular object displayed to the first observer to indicate the changing position of the moving vehicle relative to the particular object.

13. A system comprising:

one or more sensors;

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

determining respective positions of multiple observers located in different vehicles, wherein the multiple observers have respective views of an area obstructed by a barrier;

selecting, for each of the multiple observers, a proper subset of multiple objects in the area, the proper subsets each respectively being selected based on (i) a speed and direction of movement of the observer to which the proper subset corresponds and (ii) speeds and directions of movement of the multiple objects in the area, the proper subsets each including a particular object of the multiple objects;

determining, for each of the multiple observers, a position of the particular object relative to the observer;

generating, for each of the multiple observers, an indication of the position of the particular object based at least in part on the determined position of the observer; and

concurrently displaying, on a display that is located outside the vehicles and that is located between the multiple observers and the barrier, the generated indications of the position of the particular object to the respective observers such that different indications of the position of the particular object are displayed to the respective observers based on the respective positions of the multiple observers and such that the display omits, for each of the multiple observers, indications of objects in the area that are not in the proper subset selected for the observer.

15. The system of claim 13, wherein the one or more computers and one or more storage devices are further configured to perform operations comprising:

concurrently displaying, on the display, different view of the particular object to each of the multiple observers based on the determined respective positions of the multiple observers.

16. The system of claim 13, wherein the operations comprise dynamically changing the indications of the position of the particular object as the positions of the multiple observers and the particular object change.

17. The system of claim 13, wherein the display is a first display, and wherein the first display is mounted at a first side of a building that is located at an intersection of a first road and a second road, the display facing toward and extending along the first road;

wherein the system comprises a second display mounted at a second side of the building, the second display facing toward and extending along the second road;

wherein the operations comprise;

selecting, for each of the multiple observers, one or more first objects from among a set of first candidate objects that approach the intersection along the second side of the building, the one or more first objects for each of the multiple observers being selected based on a speed and location of the observer and a speed and location for the respective first candidate objects;

displaying, to each of the multiple observers and on the first display, first indications of the one or more first objects selected for the observer;

selecting, for each of multiple second observers, one or more second objects from among a set of second candidate objects that approach the intersection along the first side of the building, the one or more second objects for each of the multiple second observers being selected based on a speed and location of the second observer and a speed and location for the respective second candidate objects; and

displaying, to each of the multiple observers and on the second display, second indications of the one or more second objects selected for the second observer.

14. The system of 13, wherein

the one or more sensors comprise one or more video cameras, and

the display comprises an array of multi-directional light-emitting elements that are each configured to selectively project light in different directions.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

determining respective positions of multiple observers located in different vehicles, wherein the multiple observers have respective views of an area obstructed by a barrier;

selecting, for each of the multiple observers, a proper subset of multiple objects in the area, the proper subsets each respectively being selected based on (i) a speed and direction of movement of the observer to which the proper subset corresponds and (ii) speeds and directions of movement of the multiple objects in the area, the proper subsets each including a particular object of the multiple objects;

determining, for each of the multiple observers, a position of the particular object relative to the observer;

generating, for each of the multiple observers, an indication of the position of the particular object based at least in part on the determined position of the observer; and

concurrently displaying, on a display that is located outside the vehicles and that is located between the multiple observers and the barrier, the generated indications of the position of the particular object to the respective observers such that different indications of the position of the particular object are displayed to the respective observers based on the respective positions of the multiple observers and such that the display omits, for each of the multiple observers, indications of objects in the area that are not in the proper subset selected for the observer.