Method and apparatus for estimating a camera reference horizon

• US 20040125210A1
• Filed: 12/23/2002
• Published: 07/01/2004
• Est. Priority Date: 12/23/2002
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chen, Yang
• Granted Patent: US 7,253,835 B2
• Time in Patent Office: Days
• US Class Current: 348/207.99
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A method, apparatus, and computer program product are presented for estimating a camera reference horizon Ŷ₀. Operations include receiving images including projections of objects including horizontal edges. Vertical motion is estimated for the images and is stored. The projection of horizontal edges in the images is computed. The projected edges are segmented out and motion is compensated. Projections of the horizontal edges from image to image are used to generate tracks. Active track storage is maintained by adding data to update the storage from each subsequent image. The tracks are screened using a criteria set to produce a subset of tracks to enhance track reliability. Distances to, and heights of, the objects are then estimated. Further screening is performed by calculating an error criteria for each track and selecting a subset satisfying the error criteria. The remaining tracks are used to generate a camera reference horizon estimate Ŷ₀.

Claims

1. A method for generating a camera reference horizon estimate Ŷ

₀, the method comprising steps of;

receiving a sequence of images including projections of objects thereon, with projections including horizontal edges;

estimating vertical motion occurring in the sequence of images and storing the estimated vertical motion as a motion history;

computing the projection of the horizontal edges in the images;

segmenting the projected horizontal edges in the images;

compensating for motion in the images using the estimated vertical motion from the motion history;

tracking the projections of the horizontal edges from image to image to generate tracks representing objects and maintaining an active track storage by adding more data to update the active track storage from each subsequent image;

screening the generated tracks using screening criteria to produce a subset of tracks in order to enhance track reliability;

receiving the subset of tracks from the screening step and estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks;

further screening the subset of tracks by calculating an error criteria for each track and selecting a subset of remaining tracks satisfying the error criteria; and

generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks.

2. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 1, wherein the screening criteria used in the screening step include rejecting those tracks that are below the current reference horizon;

determining a track length for each track and keeping only those tracks that have track lengths in excess of a predetermined track length threshold;

determining the dynamic range of the tracks and keeping only those tracks that have a dynamic range exceeding a dynamic range threshold; and

determining an abnormality value for each track and ignoring tracks exceeding an abnormality threshold.

3. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 2, wherein in the step of further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

4. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 3, further comprising steps of;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

5. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 4, wherein the images include image tops, and wherein in the step of updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

6. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 5, wherein the step of estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

7. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 6, wherein in the step of generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

8. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 1, wherein in the step of further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

9. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 8, further comprising steps of;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

10. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 9, wherein the images include image tops, and wherein in the step of updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

11. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 10, wherein the step of estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

12. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 11, wherein in the step of generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

13. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 1, further comprising steps of;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

14. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 13, wherein the images include image tops, and wherein in the step of updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

15. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 14, wherein the step of estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

16. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 15, wherein in the step of generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

17. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 1, wherein the step of estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

18. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 17, wherein in the step of generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

19. A method for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 1, wherein in the step of generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

20. A computer program product for generating a camera reference horizon estimate Ŷ

₀, the computer program product comprising means, residing on a computer-readable medium, for;

receiving a sequence of images including projections of objects thereon, with projections including horizontal edges;

estimating vertical motion occurring in the sequence of images and storing the estimated vertical motion as a motion history;

computing the projection of the horizontal edges in the images;

segmenting the projected horizontal edges in the images;

compensating for motion in the images using the estimated vertical motion from the motion history;

tracking the projections of the horizontal edges from image to image to generate tracks representing objects and maintaining an active track storage by adding more data to update the active track storage from each subsequent image;

screening the generated tracks using screening criteria to produce a subset of tracks in order to enhance track reliability;

receiving the subset of tracks from the means for screening and estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks;

further screening the subset of tracks by calculating an error criteria for each track and selecting a subset of remaining tracks satisfying the error criteria; and

generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks.

21. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 20, wherein the screening criteria used by the means for screening include rejecting those tracks that are below the current reference horizon;

determining a track length for each track and keeping only those tracks that have track lengths in excess of a predetermined track length threshold;

determining the dynamic range of the tracks and keeping only those tracks that have a dynamic range exceeding a dynamic range threshold; and

determining an abnormality value for each track and ignoring tracks exceeding an abnormality threshold.

22. A computer program product for generating a camera reference horizon estimate Ŷ

₀ as set forth in claim 21, wherein in the means for further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

23. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 22, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

24. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 23, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

25. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 24, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

26. A computer program product for generating a camera reference horizon estimate Ŷ

₀ as set forth in claim 25, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

27. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 20, wherein in the means for further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

28. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 27, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

29. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 28, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

30. A computer program product for generating a camera reference horizon estimate Y, as set forth in claim 29, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

31. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 30, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

32. A computer program product for generating a camera reference horizon estimate Ŷ

₀ as set forth in claim 20, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

33. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 32, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀, is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

34. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 33, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

35. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 34, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

36. A computer program product for generating a camera reference horizon estimate Ŷ

₀ as set forth in claim 20, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

37. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 36, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

38. A computer program product for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 20, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

39. An apparatus for generating a camera reference horizon estimate Ŷ

₀, the apparatus comprising;

a memory for storing data and processing results;

a processor operationally coupled with the memory, the processor including means for receiving a sequence of images including projections of objects thereon, with projections including horizontal edges, the processor including means for performing operations thereon, for;

estimating vertical motion occurring in the sequence of images and storing the estimated vertical motion as a motion history;

computing the projection of the horizontal edges in the images;

segmenting the projected horizontal edges in the images;

compensating for motion in the images using the estimated vertical motion from the motion history;

tracking the projections of the horizontal edges from image to image to generate tracks representing objects and maintaining an active track storage by adding more data to update the active track storage from each subsequent image;

screening the generated tracks using screening criteria to produce a subset of tracks in order to enhance track reliability;

receiving the subset of tracks from the means for screening and estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks;

further screening the subset of tracks by calculating an error criteria for each track and selecting a subset of remaining tracks satisfying the error criteria; and

generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks.

40. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 39, wherein the screening criteria used by the means for screening include rejecting those tracks that are below the current reference horizon;

determining a track length for each track and keeping only those tracks that have track lengths in excess of a predetermined track length threshold;

determining the dynamic range of the tracks and keeping only those tracks that have a dynamic range exceeding a dynamic range threshold; and

determining an abnormality value for each track and ignoring tracks exceeding an abnormality threshold.

41. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 40, wherein in the means for further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

42. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 41, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

43. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 42, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

44. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 43, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

45. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 44, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

46. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 39, wherein in the means for further screening the subset of tracks, the error criteria used includes calculating a root-mean-squared fitting error for each track using the estimated distance and height, and excluding those tracks having a root-mean-squared fitting error exceeding a predetermined threshold.

47. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 46, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

48. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 47, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

49. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 48, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

50. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 49, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

51. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 39, further comprising means for;

maintaining an active camera reference horizon estimate Ŷ

₀; and

updating the active camera reference horizon estimate Ŷ

₀ with the generated camera reference horizon estimate based on the fulfillment of a criteria set.

52. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 51, wherein the images include image tops, and wherein in the means for updating the active camera reference horizon estimate Ŷ

₀ is triggered based on the fulfillment of a the criteria set including;

when a longest track exists in a number of images in the sequence of images, where the number exceeds a predetermined threshold; and

when the longest track is within a predetermined distance from an image top.

53. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 52, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

54. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 53, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

55. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 39, wherein the means for estimating distances to the objects represented by the tracks and heights of the objects represented by the tracks is performed by estimating the parameters for distance {circumflex over (D)} and height Ĥ

in the linear set,

56. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 55, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀ in the following relationship;

57. A apparatus for generating a camera reference horizon estimate Ŷ

₀, as set forth in claim 39, wherein in the means for generating a camera reference horizon estimate Ŷ

₀ from the remaining tracks, the reference horizon is determined by solving for Ŷ

₀, in the following relationship;