Systems and Methods for Using Real-Time Imagery in Navigation

• US 20170314954A1
• Filed: 05/02/2016
• Published: 11/02/2017
• Est. Priority Date: 05/02/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Golding, Andrew R., Murphy, Kevin
• Granted Patent: US 10,126,141 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G01C 21/3691 Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G08G 1/096783 where the origin of the information is a roadside individual element

Abstract

To generate navigation directions for a driver of a vehicle, a route for guiding the driver to a destination is obtained, visual landmarks corresponding to prominent physical objects disposed along the route are retrieved, and real-time imagery is collected at the vehicle approximately from a vantage point of the driver during navigation along the route. Using (i) the retrieved visual landmarks and (ii) the imagery collected at the vehicle, a subset of the visual landmarks that are currently visible to the driver is selected. Navigation directions describing the route are provided the driver, the navigation directions referencing the selected subset of the visual landmarks and excluding the remaining visual landmarks.

Claims

1. A method for generating navigation directions for drivers, the method comprising:

obtaining, by one or more processors, a route for guiding a driver of a vehicle to a destination;

retrieving, by the one or more processors from a database, a plurality of visual landmarks corresponding to prominent physical objects disposed along the route;

obtaining, by the one or more processors, real-time imagery collected at the vehicle approximately from a vantage point of the driver during navigation along the route;

using (i) the retrieved plurality of visual landmarks and (ii) the imagery collected at the vehicle, selecting, by the one or more processors, a subset of the plurality of visual landmarks that are currently visible to the driver; and

providing, to the driver, navigation directions describing the route, the navigation directions referencing the selected subset of the plurality of visual landmarks and excluding remaining ones of the plurality of visual landmarks.

2. The method of claim 1, wherein retrieving the plurality of visual landmarks includes retrieving an image of a prominent physical object and an indication of a geographic location of the prominent physical object, and wherein selecting the subset of the plurality of visual landmarks includes processing real-time imagery collected near the geographic location to determine whether the real-time imagery depicts the prominent physical object.

3. The method of claim 1, further comprising processing, by the one or more processors, the imagery collected at the vehicle using a convolutional neural network with semantic segmentation.

4. The method of claim 3, further comprising:

identifying a physical object of a certain predetermined class within a scene of the real-time imagery, using the convolutional neural network; and

comparing at least one of a size or a color of the physical object to the scene to determine whether the physical object is prominent.

5. The method of claim 1, further comprising determining lighting conditions at a current location of the vehicle, and wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks previously labeled as being suitable for the determined lighting conditions.

6. The method of claim 1, further comprising determining weather conditions at a current location of the vehicle, and wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks previously labeled as being suitable for the determined weather conditions.

7. The method of claim 1, further comprising:

prior to retrieving the plurality of visual landmarks, receiving a selection of a type of a visual landmark from the driver, the type corresponding to one of time of day or a season.

8. The method of claim 1, wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks based on a profile of the driver.

9. The method of claim 1, wherein obtaining the real-time imagery includes retrieving video feed from a camera built into the vehicle.

10. The method of claim 1, further comprising:

obtaining, by the one or more processors, an indication that the driver cannot see a visual landmark included subset of the plurality of visual landmarks; and

in response to the indication, updating a numeric metric indicative of how often the users find the visual landmark useful.

11. A system operating in a vehicle, the system comprising:

a camera configured to capture real-time imagery approximately from a vantage point of the driver;

a positioning module configured to determine a current geographic location of the vehicle;

a network interface to communicate with a server system via a communication network;

a user interface; and

processing hardware configured to;

obtain, using the captured real-time imagery and the current geographic location of the vehicle, driving directions including an instruction that references a visual landmark automatically determined as being visible in the captured real-time imagery, and

provide the instruction to the driver via the user interface.

12. The system of claim 11, wherein the processing hardware is further configured to:

receive, via the communication network, an image of a candidate visual landmark expected to be visible at the current geographic location of the vehicle,

compare the captured real-time imagery to the received image of the candidate visual landmark to determine whether the candidate visual landmark is visible in the real-time imagery, and

omit the candidate visual landmark from the instruction in response to determining that the candidate visual landmark is not visible in the real-time imagery.

13. The system of claim 11, wherein the processing hardware is further configured to:

send the captured real-time imagery to the server system via the communication network, wherein the server system determines whether a candidate visual landmark is visible in the real-time imagery, and

receive the instruction from the server system in response to the sent captured real-time imagery.

14. The system of claim 11, wherein the processing hardware is further configured to:

determine whether the captured real-time imagery depicts an object of a certain predetermined class,

determine whether the object is prominent within the real-time imagery based on at least one of size or color of the object, and

in response to determining that the object is prominent, select the object as the visual landmark.

15. The system of claim 14, wherein to determine whether the captured real-time imagery depicts an object of a certain predetermined class, the processing hardware implements a convolutional neural network with semantic segmentation.

16. The system of claim 14, wherein the certain predetermined class is one of a vehicle, a billboard, a traffic light, or a person.

17. A method for providing driving directions, the method comprising:

receiving, by one or more processors at a mobile system operating in a vehicle, a request for driving directions to a destination from a driver of the vehicle;

receiving, by the one or more processors at the mobile system, real-time imagery collected at the vehicle approximately from a vantage point of the driver;

obtaining, using the real-time imagery and a current location of the vehicle, the driving directions including an instruction that references a visual landmark automatically determined as being visible in the real-time imagery; and

providing, by the one or more processors at the mobile system, the instruction to the driver in response to the request.

18. The method of claim 17, further comprising:

selecting a candidate visual landmark that is expected to be visible at the current geographic location of the vehicle, including retrieving an image of the candidate visual landmark;

automatically processing the real-time imagery to determine whether the candidate visual landmark is visible in the real-time imagery; and

omitting the candidate visual landmark from the instruction in response to determining that the candidate visual landmark is not visible in the real-time imagery

19. The method of claim 18, wherein automatically processing the real-time imagery includes:

sending the real-time imagery from the mobile system to a server system via a communication network; and

automatically processing the real-time imagery at the sever system.

20. The method of claim 17, wherein receiving the real-time imagery includes receiving the real-time imagery from an external camera coupled to the mobile device via a short-range communication link.