Systems and methods for using real-time imagery in navigation

• US 10,126,141 B2
• Filed: 05/02/2016
• Issued: 11/13/2018
• Est. Priority Date: 05/02/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Golding, Andrew R., Murphy, Kevin
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Testardi, David A
• Application Number:
  US15/144,300
• Publication Number:
  US 20170314954A1
• Time in Patent Office: 925 Days
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G01C 21/3691 Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G08G 1/096783 where the origin of the information is a roadside individual element

Abstract

To generate navigation directions for a driver of a vehicle, a route for guiding the driver to a destination is obtained, visual landmarks corresponding to prominent physical objects disposed along the route are retrieved, and real-time imagery is collected at the vehicle approximately from a vantage point of the driver during navigation along the route. Using (i) the retrieved visual landmarks and (ii) the imagery collected at the vehicle, a subset of the visual landmarks that are currently visible to the driver is selected. Navigation directions describing the route are provided the driver, the navigation directions referencing the selected subset of the visual landmarks and excluding the remaining visual landmarks.

Claims

1. A method for generating navigation directions for drivers, the method comprising:

obtaining, by one or more processors, a route for guiding a driver of a vehicle to a destination;

retrieving, by the one or more processors from a database, a plurality of visual landmarks corresponding to prominent physical objects disposed along the route;

retrieving, by the one or more processors, a video feed from a camera mounted in the vehicle to obtain real-time imagery during navigation along the route;

using (i) the retrieved plurality of visual landmarks and (ii) the imagery collected at the vehicle, selecting, by the one or more processors, a subset of the plurality of visual landmarks that are currently visible to the driver;

providing, to the driver, navigation directions describing the route, the navigation directions referencing the selected subset of the plurality of visual landmarks and excluding remaining ones of the plurality of visual landmarks;

obtaining, by the one or more processors, an indication that the driver cannot see a visual landmark included in the subset of the plurality of visual landmarks; and

in response to the indication, updating in the database a numeric metric indicative of how often users find the visual landmark useful, wherein the numeric metric is used to select visual landmarks corresponding to prominent physical objects disposed along the route.

2. The method of claim 1, wherein retrieving the plurality of visual landmarks includes retrieving an image of a physical object and an indication of a geographic location of the physical object, and wherein selecting the subset of the plurality of visual landmarks includes processing real-time imagery collected near the geographic location to determine whether the real-time imagery depicts the physical object.

3. The method of claim 1, further comprising processing, by the one or more processors, the imagery collected at the vehicle using a convolutional neural network with semantic segmentation.

4. The method of claim 3, further comprising:

identifying a physical object of a certain predetermined class within a scene of the real-time imagery, using the convolutional neural network; and

comparing at least one of a size or a color of the physical object to the scene to determine whether the physical object is more prominent than other physical objects in the scene.

5. The method of claim 1, further comprising determining lighting conditions at a current location of the vehicle, and wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks previously labeled as being suitable for the determined lighting conditions.

6. The method of claim 1, further comprising determining weather conditions at a current location of the vehicle, and wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks previously labeled as being suitable for the determined weather conditions.

7. The method of claim 1, further comprising:

prior to retrieving the plurality of visual landmarks, obtaining a selection of a type of a visual landmark, the type corresponding to one of time of day or a season.

8. The method of claim 1, wherein retrieving the plurality of visual landmarks includes querying the database for visual landmarks based on a profile of the driver.

9. The method of claim 1, further comprising identifying, using the real-time imagery, a dynamic visual landmark for which the database stored no information, including determining that the dynamic visual landmark is visually prominent relative to the scene in the imagery based on the one or more criteria of prominence.

10. The method of claim 1, wherein obtaining the indication that the driver cannot see the visual landmark includes providing a prompt via a user interface regarding the visual landmark.

11. The method of claim 1, wherein obtaining the indication that the driver cannot see the visual landmark includes automatically determining whether the user successfully completed a maneuver to which the visual landmark is related.

12. A system operating in a vehicle, the system comprising:

a camera configured to capture real-time imagery;

a positioning module configured to determine a current geographic location of the vehicle;

a network interface to communicate with a server system via a communication network;

a user interface; and

processing hardware configured to;

determine, using the captured real-time imagery, a visual landmark that is visible to a driver of the vehicle;

obtain, using the determined visual landmark, a database that stores information related to the visual landmark, and the current geographic location of the vehicle, driving directions including an instruction that references the visual landmark,

provide the instruction to the driver via the user interface;

obtain an indication that the driver cannot see the visual landmark, and

in response to the indication, causing a numeric metric indicative of how often users find the visual landmark useful to be updated in the database, wherein the numeric metric is used to select visual landmarks corresponding to prominent physical objects disposed along the route.

13. The system of claim 12, wherein the processing hardware is further configured to:

receive, via the communication network, an image of a candidate visual landmark expected to be visible at the current geographic location of the vehicle,

compare the captured real-time imagery to the received image of the candidate visual landmark to determine whether the candidate visual landmark is visible in the real-time imagery, and

omit the candidate visual landmark from the instruction in response to determining that the candidate visual landmark is not visible in the real-time imagery.

14. The system of claim 12, wherein the processing hardware is further configured to:

send the captured real-time imagery to the server system via the communication network, wherein the server system determines whether a candidate visual landmark is visible in the real-time imagery, and

receive the instruction from the server system in response to the sent captured real-time imagery.

15. The system of claim 12, wherein the processing hardware is further configured to:

determine whether the captured real-time imagery depicts an object of a certain predetermined class,

determine whether the object is prominent within the real-time imagery based on at least one of size or color of the object, and

in response to determining that the object is prominent, select the object as the visual landmark.

16. The system of claim 15, wherein to determine whether the captured real-time imagery depicts an object of a certain predetermined class, the processing hardware implements a convolutional neural network with semantic segmentation.

17. A method for providing driving directions, the method comprising:

receiving, by one or more processors at a mobile system operating in a vehicle, a request for driving directions to a destination from a driver of the vehicle;

retrieving, by the one or more processors, a video feed from a camera mounted in the vehicle to obtain real-time imagery;

determining, by the one or more processors, a visual landmark that is visible to a driver of the vehicle in the captured real-time imagery,

obtaining, using the determined visual landmark, a database that stores information related to the visual landmark, and a current location of the vehicle, the driving directions including an instruction that references the visual landmark;

providing, by the one or more processors at the mobile system, the instruction to the driver in response to the request;

obtaining, by the one or more processors, an indication that the driver cannot see the visual landmark; and

in response to the indication, causing a numeric metric indicative of how often users find the visual landmark useful to be updated in the database, wherein the numeric metric is used to select visual landmarks corresponding to prominent physical objects disposed along the route.

18. The method of claim 17, further comprising:

selecting a candidate visual landmark that is expected to be visible at the current geographic location of the vehicle, including retrieving an image of the candidate visual landmark;

automatically processing the real-time imagery to determine whether the candidate visual landmark is visible in the real-time imagery; and

omitting the candidate visual landmark from the instruction in response to determining that the candidate visual landmark is not visible in the real-time imagery.

19. The method of claim 18, wherein automatically processing the real-time imagery includes:

sending the real-time imagery from the mobile system to a server system via a communication network; and

automatically processing the real-time imagery at the sever system.

20. The method of claim 17, wherein receiving the real-time imagery includes receiving the real-time imagery from an external camera coupled to the mobile device via a short-range communication link.