Augmented reality navigation systems and methods

• US 10,495,476 B1
• Filed: 01/24/2019
• Issued: 12/03/2019
• Est. Priority Date: 09/27/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Phiar Technologies, Inc. (Alphabet Inc.)
• Inventors: Yu, Chen-Ping, Li, Xinyu, Chandrasekhar, Aaditya
• Primary Examiner(s): To, Tuan C
• Application Number:
  US16/255,825
• Time in Patent Office: 313 Days
• Field of Search: 701431
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01C 21/3691 Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

A system and method may generate a more realistic augmented reality (AR) overlay by generating a segmentation image and blending it with one or more other images. The system may generate a segmentation image based on an input image. The segmentation image may be blended with an AR path overlay image to generate an object-masked AR path overlay image. The object-masked AR path overlay image may be blended with the input image to generate an output image.

Claims

1. A method, comprising:

determining a guidance route;

generating a guidance indicator corresponding to the guidance route;

obtaining an image of the guidance route including an object;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the object in the image, the first pixels having a first common value, and second pixels having a second common value different from the first common value;

modifying a portion of the guidance indicator based on at least a position of the first pixels in the segmentation image; and

displaying the modified guidance indicator on a display.

2. The method of claim 1, wherein generating the segmentation image is performed by applying semantic segmentation on the image.

3. The method of claim 2, wherein the image comprises an image of a roadway, and wherein the at least one object comprises at least one object on the roadway.

4. The method of claim 3, wherein generating the guidance indicator associated with the navigational guidance route comprises generating an augmented-reality path overlay image including the guidance indicator.

5. The method of claim 4, wherein modifying the guidance indicator based on the at least one object in the image comprises modifying pixels in the augmented-reality path overlay image that spatially correspond to the first pixels having the first common value in the segmentation image.

6. The method of claim 5, wherein displaying the modified guidance indicator in association with the navigational guidance route comprises generating an output image by combining the image with the augmented-reality path overlay image having the modified guidance indicator.

7. The method of claim 6, further comprising displaying the output image on a display of a mobile electronic device.

8. The method of claim 5, wherein displaying the modified guidance indicator in association with the navigational guidance route comprises projecting the augmented-reality path overlay image having the modified guidance indicator onto a windshield of a vehicle.

9. The method of claim 1, further comprising determining the navigational guidance route based on a global positioning system location and map information.

10. A navigation system comprising one or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors to perform acts comprising:

determining a guidance route;

obtaining an image of the guidance route including an object;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the object in the image, the first pixels having a first common value, and second pixels having a second common value different from the first common value;

modifying a portion of the guidance indicator based on at least a position of the first pixels in the segmentation image; and

displaying the modified guidance indicator on a display.

11. The navigation system of claim 10, wherein generating the segmentation image is performed by applying semantic segmentation on the image.

12. The navigation system of claim 11, wherein the image comprises an image of a roadway, and wherein the at least one object comprises at least one object on the roadway.

13. The navigation system of claim 12, wherein generating the guidance indicator associated with the navigational guidance route comprises generating an augmented-reality path overlay image including the guidance indicator.

14. The navigation system of claim 13, wherein modifying the guidance indicator based on the at least one object in the image comprises modifying pixels in the augmented-reality path overlay image that spatially correspond to the first pixels having the first common value in the segmentation image.

15. The navigation system of claim 14, wherein displaying the modified guidance indicator in association with the navigational guidance route comprises generating an output image by combining the image with the augmented-reality path overlay image having the modified guidance indicator.

16. The navigation system of claim 15, further comprising instructions for displaying the output image on a display of a mobile electronic device.

17. The navigation system of claim 16, wherein displaying the modified guidance indicator in association with the navigational guidance route comprises projecting the augmented-reality path overlay image having the modified guidance indicator onto a windshield of a vehicle.

18. A non-transitory computer readable medium storing instructions that, when executed by a processor cause the processor to perform acts comprising:

determining a guidance route;

generating a guidance indicator corresponding to the guidance route;

obtaining an image of the guidance route including an object;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the object in the image, the first pixels having a first common value, and second pixels having a second common value different from the first common value;

modifying a portion of the guidance indicator based on at least a position of the first pixels in the segmentation image; and

displaying the modified guidance indicator on a display.

19. The non-transitory computer readable medium of claim 18, wherein generating the segmentation image is performed by applying semantic segmentation on the image.

20. The navigation system of claim 18, wherein the image comprises an image of a roadway, and wherein the at least one object comprises at least one object on the roadway.