Visual odometry for low illumination conditions using fixed light sources

• US 10,127,461 B2
• Filed: 08/12/2016
• Issued: 11/13/2018
• Est. Priority Date: 08/12/2016
• Status: Active Grant

• Current Assignee: HERE Global B.V. (HERE Holding Corporation)
• Original Assignee: HERE Global B.V. (HERE Holding Corporation)
• Inventors: Viswanathan, Anirudh
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US15/235,326
• Publication Number:
  US 20180047147A1
• Time in Patent Office: 823 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/56 provided with illuminating means

Abstract

First and second image data is captured comprising a first and second image, respectively. A fixed light source is identified in each of the first and second images. A first ground plane is determined in the first image data. A first (second) intersection is determined, wherein the first (second) intersection is a point in the first image where a virtual lamp post corresponding to the fixed light source in the first (second) image intersects with the first (second) ground plane. The first image data and the second image data are transformed to provide a first and second inverse perspective map (IPM) comprising a first transformed intersection and a second transformed intersection, respectively. Movement parameters are determined based on the location of the first transformed intersection in the first IPM and the location of the second transformed intersection in the second IPM.

Claims

1. A method comprising:

capturing first image data comprising a first image, the first image captured by an imaging device;

identifying a fixed light source from the first image data;

determining a first ground plane in the first image data;

determining a first intersection wherein the first intersection is a point in the first image where a virtual lamp post corresponding to the fixed light source in the first image intersects with the first ground plane, wherein the virtual lamp post extends vertically from the fixed light source to the first ground plane;

capturing second image data comprising a second image, the second image captured by the imaging device, wherein the second image sufficiently overlaps with the first image;

identifying the fixed light source in the second image data;

determining a second ground plane in the second image data;

determining a second intersection, the second intersection is the point in the second image where a virtual lamp post corresponding to the fixed light source in the second image data intersects with the second ground plane;

transforming the first image data and the second image data to obtain a first inverse perspective map comprising a first transformed intersection and a second inverse perspective map comprising a second transformed intersection; and

based at least in part on the first transformed intersection and the second transformed intersection, determining one or more movement parameters related to movement of the imaging device between the capturing of the first image data and the capturing of the second image data.

2. A method according to claim 1, wherein the movement parameters comprise at least one selected from the group of (a) one or more translation parameters, (b) one or more rotation parameters, (c) a distance traveled, (d) a position, (e) a speed, (f) a velocity, and (g) a heading.

3. A method according to claim 1, wherein the first ground plane is applied as the second ground plane.

4. A method according to claim 1, wherein the imaging device is affixed to a mode of transportation at either a fixed height, a known height, or both.

5. A method according to claim 1, wherein the first and/or second ground plane is determined by computing a homography based at least in part on the first and/or second image data.

6. A method according to claim 1, wherein the height of the imaging device is determined based on either a length of a lane marking, the distance between consecutive lane markings, width of a lane between a pair of lane markings, or combination thereof based on an image captured by the imaging device.

7. A method according to claim 1, wherein the first and/or second ground plane is determined using gradients in a color and/or intensity profile of the first or second image data, changes in the color and/or intensity profile of the first or second image data, or a combination thereof.

8. A method according to claim 1, wherein the first and/or second ground plane is determined based at least in part on one or more local road features identified in at least the first or second image data and another set of captured image data.

9. A method according to claim 1, wherein the first and/or second ground plane is determined based at least in part on one or more objects identified in the first and/or second image.

10. A method according to claim 1, wherein a slope of the first and/or second ground plane is inferred based at least in part on an illumination profile caused by a light source associated with a mode of transportation to which the imaging device is affixed reflecting off a surface corresponding to the first or second ground plane.

11. A method according to claim 1, wherein a slope of the first and/or second ground plane is inferred based on the position of one or more lane markings within the first or second image data.

12. A method according to claim 1, further comprising

performing a full reconstruction of a surface using dense optical flow techniques for areas illuminated by a light source associated with a mode of transportation to which the imaging device is affixed,

wherein the surface corresponds to the first or second ground plane.

13. A method according to claim 1, wherein measurements by one or more sensors associated with a mode of transportation to which the imaging device is affixed captured in real- or near real-time with the capturing of the first or second image data are used to determine a slope of the first and/or second ground plane.

14. A method according to claim 1, wherein the movement parameters are determined in real-time or near real-time with respect to the capturing of the second image data.

18. A method according to claim 1, wherein the movement parameters comprise at least one selected from the group of (a) one or more translation parameters, (b) one or more rotation parameters, (c) a distance traveled, (d) a position, (e) a speed, (f) a velocity, and (g) a heading.

15. An apparatus comprising at least one processor and at least one memory storing computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:

capture first image data comprising a first image, the first image captured by an imaging device;

identify a fixed light source from the first image data;

determine a first ground plane in the first image data;

determine a first intersection wherein the first intersection is a point in the first image where a virtual lamp post corresponding to the fixed light source in the first image intersects with the first ground plane, wherein the virtual lamp post extends vertically from the fixed light source to the first ground plane;

capture second image data comprising a second image, the second image captured by the imaging device, wherein the second image sufficiently overlaps with the first image;

identify the fixed light source in the second image data;

determine a second ground plane in the second image data;

determine a second intersection, the second intersection is the point in the second image where a virtual lamp post corresponding to the fixed light source in the second image data intersects with the second ground plane;

transform the first image data and the second image data to obtain a first inverse perspective map comprising a first transformed intersection and a second inverse perspective map comprising a second transformed intersection; and

based at least in part on the first transformed intersection and the second transformed intersection, determine one or more movement parameters related to movement of the imaging device between the capturing of the first image data and the capturing of the second image data.

16. An apparatus according to claim 15, wherein the movement parameters are determined in real-time or near real-time with respect to the capturing of the second image data.

17. An apparatus according to claim 15, wherein to determine the first ground plane, the at least one memory and the computer program code are configured to, with the processor, cause the apparatus to at least compute a homography based at least in part on the first image data.

19. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:

cause first image data comprising a first image to be captured, the first image captured by an imaging device;

identify a fixed light source from the first image data;

determine a first ground plane in the first image data;

determine a first intersection wherein the first intersection is a point in the first image where a virtual lamp post corresponding to the fixed light source in the first image intersects with the first ground plane, wherein the virtual lamp post extends vertically from the fixed light source to the first ground plane;

cause second image data comprising a second image to be captured, the second image captured by the imaging device, wherein the second image sufficiently overlaps with the first image;

identify the fixed light source in the second image data;

determine a second ground plane in the second image data;

determine a second intersection, the second intersection is the point in the second image where a virtual lamp post corresponding to the fixed light source in the second image data intersects with the second ground plane;

transform the first image data and the second image data to obtain a first inverse perspective map comprising a first transformed intersection and a second inverse perspective map comprising a second transformed intersection; and

based at least in part on the first transformed intersection and the second transformed intersection, determine one or more movement parameters related to movement of the imaging device between the capturing of the first image data and the capturing of the second image data.

20. A method comprising:

capturing first image data comprising a first image, the first image captured by an imaging device on board a mode of transportation;

based at least in part on a fixed light source in the first image data and a first ground plane in the first image data, determining a first intersection in the first image;

capturing second image data comprising a second image, the second image captured by an imaging device;

based at least in part on the fixed light source in the second image data and a second ground plane in the second image data, determining a second intersection in the second image;

transforming the first image data and the second image data to obtain a first inverse perspective map comprising a first transformed intersection and a second inverse perspective map comprising a second transformed intersection;

based at least in part on the first transformed intersection and the second transformed intersection, determining one or more movement parameters related to movement of the mode of transportation between the capturing of the first image data and the capturing of the second image data; and

providing a communication comprising at least one of the one or more movement parameters to a remote computing entity.

21. The method of claim 20, wherein the communication comprises an identifier configured to indicate that the at least one of the one or more movement parameters were determined using visual odometry or low illumination condition visual odometry.