Sequential rolling bundle adjustment

• US 9,177,384 B2
• Filed: 07/31/2013
• Issued: 11/03/2015
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Trimble Navigation Limited (Trimble Inc.)
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: Weisenburger, Shawn D.
• Primary Examiner(s): Liu, Li
• Application Number:
  US13/954,995
• Publication Number:
  US 20150036888A1
• Time in Patent Office: 825 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A method for estimating position and orientation of an image-capturing device is proposed. The method comprises the step of obtaining a preceding set of frames by using the image-capturing device. Each frame includes a set of image data. The method of the present technology further comprises the step of estimating a previous position and orientation of the image-capturing device by using the set of image data included in at least one preceding frame, and the step of estimating a current position and orientation of the image-capturing device by replacing a set of image data included in at least one preceding frame by a set of image data included in at least one subsequent frame. At least one subsequent frame is obtained by using the image-capturing device.

Claims

1. A method of estimating position and orientation of an image-capturing device comprising:

(A) obtaining a set of images by using said image-capturing device by processing a preceding set of frames;

each said frame including a set of image data;

(B) estimating previous position and orientation states of said image-capturing device by using state and covariance matrices including a set of image data included in at least one said preceding frame; and

(C) estimating current position and orientation states of said image-capturing device by performing a sequential update of said state and covariance matrices;

wherein for each new frame, said sequential update of said state and covariance matrices is performed by removing position and orientation states associated with an oldest preceding frame and by adding position and orientation states for said new frame.

2. The method of claim 1, wherein said step (A) further comprises:

(A1) selecting said image capturing device from the group consisting of;

a digital camera;

a digital video camera;

a digital camcorder;

a stereo digital camera;

a stereo video camera;

a motion picture camera; and

a television camera.

3. The method of claim 1, wherein said step (A) further comprises:

(A2) calibrating said image-capturing device to correct the images for lens distortions;

said lens distortions selected from the group consisting of;

a principal distance distortion;

a focal length;

a principal point offset;

a radial distortion; and

a tangential distortion.

4. The method of claim 1, wherein said step (B) further comprises:

(B1) making an initial estimate of said position and orientation states associated with said current new frame by using an estimate of position and orientation states associated with a previous frame.

5. The method of claim 4, wherein said step (B1) further comprises:

(B1,

     1) identifying key-points by using 3 non-collinear targets with known absolute or relative positions.

6. The method of claim 4, wherein said step (B1) further comprises:

(B1, 2 ) identifying a set of at least five key-points associated with said at least one preceding frame;

identifying said set of at least five key-points associated with at least one said subsequent frame; and

based on said set of at least five key-points being identified in said at least one preceding frame and in said at least one subsequent frame substantially accurately determining a relative change between said at least one preceding frame and said at least one subsequent frame.

7. The method of claim 1, wherein said step (B) further comprises:

(B2) for each new frame, estimating initial position and orientation states of said image-capturing device by using an additional navigation device.

8. The method of claim 7, wherein said step (B2) further comprises:

(B2,

     1) selecting said additional navigation device from the group consisting of;

a smart phone;

a tablet;

a laptop; and

a personal navigation device.

9. The method of claim 1, wherein said step (C) further comprises:

(C1) for each new frame, sequentially updating said state and covariance matrices by replacing a set of image data related to at least five key-points included in said preceding frame by a set of image data related to at least five same key-points included in said subsequent frame.

10. The method of claim 1, wherein said step (C) further comprises:

(C2) for each new frame, sequentially updating said state and covariance matrices;

wherein the only new inversion required for said sequential update is an inversion of a square matrix with the dimensions equal to the number of new measurements associated with said new frame.

11. The method of claim 1, wherein said step (C) further comprises:

(C3) estimating current position and orientation states of said image-capturing device by using the method of Kalman filtering.

12. The method of claim 1 further comprising:

(D) selecting a relevant coordinate system.

13. The method of claim 12, wherein said step (D) further comprises:

(D1) selecting said relevant coordinate system from the group consisting of;

a Global Coordinate System; and

a Local Coordinate System.

14. An apparatus for estimation of position and orientation states of an image-capturing device comprising:

(A) a means for obtaining a set of images by processing a preceding set of frames;

each said frame including a set of image data;

(B) a means for estimating previous position and orientation states of said image-capturing device by using state and covariance matrices including a set of image data included in at least one said preceding frame; and

(C) a means for estimating current position and orientation states of said image-capturing device by performing a sequential update of said state and covariance matrices;

wherein for each new frame, said sequential update of said state and covariance matrices is performed by removing position and orientation states associated with an oldest preceding frame and by adding position and orientation states for said new frame.

15. The apparatus of claim 14, wherein said means (A) further comprises:

(A1) said image capturing device selected from the group consisting of;

a digital camera;

a digital video camera;

a digital camcorder;

a stereo digital camera;

a stereo video camera;

a motion picture camera; and

a television camera.

16. The apparatus of claim 14, wherein said means (B) further comprises:

(B1) an algorithm including at least the step of making an initial estimate of said position and orientation states associated with said current new frame by using an estimate of position and orientation states associated with a previous frame.

17. The apparatus of claim 14, wherein said means (B) further comprises:

(B2) an algorithm including at least the step of identifying key-points by using 3 non-collinear targets with known absolute or relative positions.

18. The apparatus of claim 14, wherein said means (B) further comprises:

(B2) an algorithm including at least the step of identifying a set of at least five key-points associated with said at least one preceding frame;

identifying said set of at least five key-points associated with at least one said subsequent frame; and

based on said set of at least five key-points being identified in said at least one preceding frame and in said at least one subsequent frame substantially accurately determining a relative change between said at least one preceding frame and said at least one subsequent frame.

19. The apparatus of claim 14, wherein said means (B) further comprises:

(B3) an additional navigation device configured for each new frame to estimate initial position and orientation states of said image-capturing device.

20. The apparatus of claim 19, wherein said additional navigation device is selected from the group consisting of:

a smart phone;

a tablet;

a laptop; and

a personal navigation device.

21. The apparatus of claim 14, wherein said means (C) further comprises:

(C1) an algorithm, for each new frame including at least the step of sequentially updating said state and covariance matrices by replacing a set of image data related to at least five key-points included in said preceding frame by a set of image data related to at least five same key-points included in said subsequent frame.

22. The apparatus of claim 14, wherein said means (C) further comprises:

(C2) an algorithm, for each new frame including at least the step of sequentially updating said state and covariance matrices;

wherein the only new inversion required for said sequential update is an inversion of a square matrix with the dimensions equal to the number of new measurements associated with said new frame.

23. The apparatus of claim 14, wherein said means (C) further comprises:

(C2) Kalman filtering algorithm.

24. An article comprising a non-transient storage medium having stored thereon instructions that, when executed, result in a computing platform having the capability to perform an estimation of position and orientation of an image-capturing device by executing an algorithm further comprising:

(A) obtaining a set of images by using said image-capturing device by processing a preceding set of frames;

each said frame including a set of image data;

(B) estimating previous position and orientation states of said image-capturing device by using state and covariance matrices including a set of image data included in at least one said preceding frame; and

(C) estimating current position and orientation states of said image-capturing device by performing a sequential update of said state and covariance matrices;

wherein for each new frame, said sequential update of said state and covariance matrices is performed by removing position and orientation states associated with an oldest preceding frame and by adding position and orientation states for said new frame.