Method to generate virtual display surfaces from video imagery of road based scenery

• US 9,135,754 B2
• Filed: 05/03/2013
• Issued: 09/15/2015
• Est. Priority Date: 05/07/2012
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ng-Thow-Hing, Victor, Sridhar, Srinath
• Primary Examiner(s): Chu, David H
• Application Number:
  US13/887,155
• Publication Number:
  US 20130293582A1
• Time in Patent Office: 865 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Generating a virtual model of environment in front of a vehicle based on images captured using an image capturing. The Images captured on an image capturing device of a vehicle are processed to extract features of interest. Based on the extracted features, a virtual model of the environment is constructed. The virtual model includes one or more surfaces. Each of the surfaces may be used as a reference surface to attach and move graphical elements generated to implement augmented reality (AR). As the vehicle moves, the graphical elements move as if the graphical elements are affixed to the one of the surfaces. By presenting the graphical elements to move together with real objects in front of the vehicle, a driver perceives the graphical elements as being part of the actual environment and reduces distraction or confusion associated with the graphical elements.

Claims

1. A method, comprising:

capturing an image by an image capturing device in a vehicle;

detecting visual features in the image indicative of a position of a road relative of the vehicle by at least;

cropping the captured image to extract a portion of image including outline of a road,

performing inverse perspective mapping on the cropped image to generate a bird'"'"'s eye view image,

using a separable filter on the bird'"'"'s eye view image to generate a filtered image,

detecting line features representing edges of the road from the filtered image, and

fitting curves onto the detected line features;

constructing a virtual model of environment in front of the vehicle based on the detected visual features, the virtual model including at least one surface;

generating graphical elements to convey information on a display device in the vehicle;

configuring the generated graphical elements with reference to the at least one surface of the virtual model; and

displaying the configured graphical elements on the display device.

2. The method of claim 1, wherein the display device is a heads-up display device projecting the graphical elements on a windshield of the vehicle.

3. The method of claim 1, wherein the virtual model represents a trench including a left wall, a right wall, a floor and a ceiling.

4. The method of claim 1, wherein the configured graphical elements move in conjunction with the at least one surface to generate an appearance that the configured graphical elements are fixed to the at least one surface.

5. The method of claim 1, wherein generating the graphical elements comprises:

determining location and heading of the vehicle; and

retrieving information from an information source based on the location or heading of the vehicle.

6. The method of claim 5, wherein generating the graphical elements further comprises using a profile of a user and the location or heading of the vehicle to generate the graphical elements.

7. The method of claim 1, further comprising:

determining heading and location of the vehicle,

determining current time;

determining location of sun relative to the vehicle based on the determined current time, the determined heading and the determined location;

generating shadows of at least one of the configured graphical elements based on the location of the sun; and

displaying the shadows on the display device.

8. The method of claim 1, further comprising:

detecting a pedestrian likely to cross the road in front of the vehicle; and

generating a graphical element representing a crosswalk at a portion of the road across which the pedestrian is likely to cross the road.

9. An augmented reality system comprising:

an image capturing device installed in a vehicle to capture an image comprising a road in front of the vehicle;

an image feature detector coupled to the image capturing device and configured to detect visual features in the image indicative of a position of a road relative to the vehicle by at least;

cropping the captured image to extract a portion of image including outline of a road; and

performing inverse perspective mapping on the cropped image to generate a bird'"'"'s eye view image,

using a separable filtering on the bird'"'"'s eye view image to generate a filtered image,

detecting line features representing edges of the road image from the filtered image, and

fitting curves onto the detected line features; and

a model generator coupled to the image feature detector and configured to construct a virtual model of environment in front of the vehicle based on the visual features, the virtual model including at least one surface;

a graphical element generator configured to graphical elements to convey information;

a graphical image renderer configured to set configurations of the generated graphical elements with reference to the at least one surface of the virtual model; and

a display device coupled to the graphical image renderer and configured to display the generated graphical elements according to the set configurations.

10. The system of claim 9, wherein the display device is a heads-up display device projecting the graphical elements on a windshield of the vehicle.

11. The system of claim 9, wherein the virtual model represents a trench including a left wall, a right wall, a floor and a ceiling.

12. The system of claim 9, wherein the configurations of the graphical elements are set to move the graphical elements in conjunction with the at least one surface to generate an appearance that the graphical elements are fixed to the at least one surface.

13. The system of claim 9, wherein the graphical element generator is configured to:

determine location and heading of the vehicle; and

retrieve information from an information source based on the location or heading of the vehicle.

14. The system of claim 9, further comprising an environment state analyzer configured to:

determine location of sun based on the determined current time, the determined heading and the determined location; and

generate shadows of at least one of the configured graphical elements based on the location of the sun for display on the display device.

15. The system of claim 9, further comprising a pedestrian detector configured to detect a pedestrian likely to cross the road in front of the vehicle, and wherein the graphical elements generator is further configured to generate a graphical element representing a crosswalk at a portion of the road across which the pedestrian is likely to cross the road.

16. A non-transitory computer-readable storage medium storing instructions thereon, the instructions when executed by a processor causes the processor to:

detect visual features in an image indicative of a position of a road relative to the vehicle, the image captured by an image capturing device in the vehicle by at least;

cropping the captured image to extract a portion of image including outline of a road,

performing inverse perspective mapping on the cropped image to generate a bird'"'"'s eye view image,

using a separable filter on the bird'"'"'s eye view image to generate a filtered image,

detecting line features representing edges of the road from the filtered image, and

fitting curves onto the detected line features;

construct a virtual model of environment in front of the vehicle based on the visual features, the virtual model including at least one surface;

generate graphical elements to convey information on a display device in the vehicle;

configure the generated graphical elements with reference to the at least one surface of the virtual model; and

display the configured graphical elements on the display device.