Method and System for Recovery of 3D Scene Structure and Camera Motion From a Video Sequence

• US 20140147032A1
• Filed: 11/11/2011
• Published: 05/29/2014
• Est. Priority Date: 11/09/2010
• Status: Active Grant

• Current Assignee: The Provost, Fellows, Foundation Scholars, and other members of Board, of the College of the Holy and Undivided Trinity of Queen Elizabeth near Dublin
• Original Assignee: The Provost Fellows and Scholars of the College of the Holy and Undivided Trinity of Queen Elizabeth, near Dublin
• Inventors: Lacey, Gerard, Yous, Sofiane, Redmond, Peter
• Granted Patent: US 9,196,054 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An improved method and a system are disclosed for recovering a three-dimensional (3D) scene structure from a plurality of two-dimensional (2D) image frames obtained from imaging means. Sets of 2D features are extracted from the image frames, and sets corresponding to successive image frames are matched, such that at least one pair of matched 2D features refers to a same 3D point in a 3D scene captured in 2D in the image frames. A 3D ray is generated by back-projection from each 2D feature, and the generated 3D rays are subjected to an anchor-based minimization process, for determining camera motion parameters and 3D scene points coordinates, thereby recovering a structure of the 3D scene.

Claims

1. A method of recovering a 3D scene structure from a plurality of image frames obtained from imaging means, the method comprising the steps of

extracting a first set of 2D features from a first image frame,

extracting a second set of 2D features from a second image frame,

matching the second set with the first set, such that at least one pair of matched 2D features refers to a same 3D point in a 3D scene,

generating a 3D ray by back-projection from each 2D feature, and

subjecting the generated 3D rays to an anchor-based minimization process, for determining camera motion parameters and 3D scene points coordinates, thereby recovering a structure of said 3D scene.

2. A method according to claim 1, comprising the further step of processing the determined camera motion parameters using 3D ray intersection according to the function

3. A method according to claim 2, wherein the anchor-based minimization process comprises a step of selecting a 3D point in the 3D scene as an anchor, being a centre of rotation, whereby camera motion is expressed as a convolution of an initial motion that transform a 3D ray from the first image frame into a corresponding view in the second image frame, and a rotation around the selected anchor.

4. A method according to claim 1, wherein the initial motion is one or any combination of a pure translation, a pure rotation and a pure scaling.

5. A method according to claim 1, comprising the further step of processing the amount of pure translation of the camera for bringing the 3D ray connecting the camera to the anchor for intersecting with the image plan at I₂, according to the function

Ti=(S0*(x_I2),S0*(y_I2−

y_I1),0)

wherein;

S0;

The z-coordinate of the Anchor A which also the initial scale.

(x_I1,y_I1) and (x_I2,y_I2);

The coordinates of I₁ and I₂ in the local coordinates system.

6. A method according to claim 3, wherein the minimization-based motion parameters estimation algorithm comprises the steps of:

selecting a 2D feature, I₁(x_I1,y_I1), in the first image frame and its corresponding 2D feature I₂(x_I2,y_I2) in the next image frame, to be the anchor A,

initialising the camera motion parameter, randomly if the process loop is a first iteration, or with the camera motion parameter from the previous frame if the process loop is a subsequent iteration,

applying a gradient descent minimization over a rotation matrix R and a scale to be determined S to minimize the error;

7. A system for recovering a 3D scene structure from a plurality of image frames obtained from imaging means, said system comprising

the imaging means and

data processing means interfaced with the imaging means, and configured to extract a set of 2D features from each image frame,

match each set with a set extracted from the previous image frame, such that at least one pair of matched 2D features refers to a same 3D point in a 3D scene,

generate a 3D ray by back-projection from each 2D feature, and

subject the generated 3D rays to an anchor-based minimization process, for determining camera motion parameters and 3D scene points coordinates, thereby recovering a structure of said 3D scene.

8. A system according to claim 7, wherein the imaging means is selected from the group comprising one or more analogue cameras, digital cameras, stereoscopic cameras, plenoptic cameras, networked cameras.

9. A system according to claim 7, wherein the imaging means is interfaced to the data processing means with a wireless data link.

10. A system according to claim 7, embodied in the onboard navigation system of a large complex vehicle such as an aircraft, land transport vehicle or sea transport vehicle.

11. A system according to claim 7, embodied in the flight management system of an airliner.

12. A set of instructions recorded on a data carrying medium which, when processed by a data processing terminal connected to imaging means and obtaining image frames therefrom, configures the terminal to perform the steps of

extracting a first set of 2D features from a first image frame,

extracting a second set of 2D features from a second image frame,

matching the second set with the first set, such that at least one pair of matched 2D features refers to a same 3D point in a 3D scene,

generating a 3D ray by back-projection from each 2D feature, and

subjecting the generated 3D rays to an anchor-based minimization process, for determining camera motion parameters and 3D scene points coordinates, thereby recovering a structure of said 3D scene.

13. A set of instructions according to claim 12, wherein the data processing terminal is onboard a vehicle.

14. A set of instructions according to claim 12, embodied in an aircraft user application, wherein the data processing terminal comprises an aircraft flight management system.

15. A user interface for use with the system of claim 7 or output by the set of instructions of claim 12, to which the recovered structure of the 3D scene is output.

16. A user interface according to claim 15, further comprising at least one non-moving reference indicator, located substantially at the centre of the user interface, and at least one reference indicator corresponding to an environmental feature and/or hazard determined as a 3D point from the camera image data, which the user interface translates and/or rotates relative to the at least one non-moving reference indicator in use.

17-19. -19. (canceled)