Camera pose estimation apparatus and method for augmented reality imaging

• US 8,452,080 B2
• Filed: 05/22/2008
• Issued: 05/28/2013
• Est. Priority Date: 05/22/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Engedal, Torbjrn
• Primary Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US12/601,188
• Publication Number:
  US 20100232727A1
• Time in Patent Office: 1,832 Days
• Field of Search: 382/154, 382/100, 382/107, 382/103, 382/104, 382/209, 382/118, 382/291, 382/159, 382/285, 382/190, 382/286, 382/155, 382/276, 382/181, 345/419, 345/418, 348/169, 348/E5.024, 702/127, 702/153, 702/150, 702/152, 700/245, 700/90, 715/863, 715/700
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

An apparatus for providing an estimate for a 3D camera pose relative to a scene from 2D image data of 2D image frame provided by said camera. A candidate 2D key points detector determines candidate 2D key points from the 2D image frame. A detected 3D observations detector determines detected 3D observations from pre-recorded scene data and the candidate 2D key points. A detected 3D camera pose estimator determines a detected 3D camera pose estimate from the camera data, the detected 3D observations and the candidate 2D key points. A first storage stores the detected 2D candidate key points and the 2D image data, and outputs in response to a 3D camera pose estimate output previous 2D image data and candidate 2D key points related to a previous 3D camera pose estimate output. A second storage stores and outputs a previous 3D camera pose estimate. A tracked 3D observations detector determines tracked 3D observations from the 2D image data, the candidate 2D key points, the camera data, the previous 2D image data and candidate 2D key points, the previous 3D camera pose estimate and 3D scene model data. A pose estimate selector outputs a selected one of the detected camera pose estimate and the previous 3D camera pose estimate. A 3D camera pose estimator computes and outputs the 3D camera pose estimate from the camera data, the detected 3D observations, the tracked 3D observations and the selected 3D camera pose estimate.

Claims

1. An apparatus for providing an estimate for a 3D camera pose relative to a scene from 2D image data of a 2D image frame provided by said camera, the apparatus comprising:

(a) a candidate 2D key points detector having inputs for receiving camera data from a camera data source and the 2D image data, and an output for providing candidate 2D key points;

(b) a detected 3D observations detector having inputs for receiving pre-recorded scene data and the candidate 2D key points, and an output for providing detected 3D observations;

(c) a detected 3D camera pose estimator having inputs for receiving the camera data, the detected 3D observations and the candidate 2D key points, and an output for providing a detected 3D camera pose estimate;

(d) a 2D image data and candidate 2D key points storage having inputs for receiving the detected 2D candidate key points and the 2D image data, and a control input for receiving a control signal related to a 3D camera pose estimate output, so as to store the 2D image data and the candidate 2D key points related to the 3D camera pose estimate output and to output a previous 2D image data and candidate 2D key points related to a previous 3D camera pose estimate output;

(e) a previous 3D camera pose estimate storage having an input for receiving the 3D camera pose estimate output, and adapted to store the 3D camera pose estimate output and to output the said previous 3D camera pose estimate;

(f) a tracked 3D observations detector having inputs for receiving the 2D image data, the candidate 2D key points, the camera data, the previous 2D image data and candidate 2D key points, the previous 3D camera pose estimate and 3D scene model data from a 3D scene model data source, and an output for providing tracked 3D observations;

(g) a pose estimate selector having inputs for receiving the detected camera pose estimate and the said previous 3D camera pose estimate, and adapted to output as a selected 3D camera pose estimate one of the said detected 3D camera pose estimate and the said previous 3D camera pose estimate; and

(h) a 3D camera pose estimator having inputs for receiving the camera data, the detected 3D observations, the tracked 3D observations and the selected 3D camera pose estimate, and an output for providing the 3D camera pose estimate output, wherein the detectors, the estimators and the selector are implemented with software executed by one or more processing units.

2. The apparatus of claim 1, wherein the receiving pre-recorded scene data comprises a pretrained 2D key points classification tree or a 2D key points data base.

3. The apparatus of claim 2, wherein the pretrained 2D key points classification tree or the 2D key points data base is adapted to link the 2D key points to 3D coordinates of the 2D key points, wherein the 3D coordinates are referenced to a coordinate system of the 3D scene model.

4. The apparatus of claim 1, wherein the camera data includes a camera lens focal length and a camera principal point.

5. The apparatus of claim 1, wherein the detected 3D camera pose estimator is adapted to output with the detected 3D camera pose estimate a first estimate quality indication, the 3D camera pose estimator is adapted to output with the 3D camera pose estimate a second quality indication, and the 3D camera pose selector is adapted to output the selected 3D camera pose estimate in response to the first and second estimate quality indications.

6. The apparatus of claim 1, so adapted that the previous 2D image data, the previous candidate 2D key points and the previous 3D camera pose output all relate to a previous 2D image frame immediately preceding the said 2D image frame.

7. The apparatus of claim 1, wherein the detected 3D observations detector and the tracked 3D observations detector are adapted to determine scores for the detected 3D observations and scores for the tracked 3D observations, and to output detected 3D observations scores and tracked 3D observation scores with the detected 3D observations and said tracked 3D observations, respectively.

8. The apparatus of claim 7, wherein the 3D camera pose estimator is adapted to

(a) normalize the detected 3D observations scores and the tracked 3D observation scores, and

(b) provide selected ones of the detected 3D observations and tracked 3D observations in response to normalized scores thereof.

9. The apparatus of claim 1, wherein the 3D camera pose estimator is adapted to determine a validity of any of the selected 3D camera pose estimate being input to the 3D camera pose estimator, and to maintain as its 3D camera pose estimate a previous 3D camera pose estimate when the selected 3D camera pose estimate being input is determined not valid.

10. A non-transitory computer-readable medium comprising instructions stored thereon to cause one or more processing units to perform operations to provide an estimate for a 3D camera pose relative to a scene from 2D image data of a 2D image frame, the operations comprising:

providing candidate 2D key points based on camera data from a camera data source and the 2D image data;

providing detected 3D observations based on pre-recorded scene data and the candidate 2D key points;

providing a detected 3D camera pose estimate based on the camera data, the detected 3D observations and the candidate 2D key points;

receiving a control signal related to a 3D camera pose estimate output;

storing the 2D image data and the candidate 2D key points related to the 3D camera pose estimate output, and providing a previous 2D image data and candidate 2D key points related to a previous 3D camera pose estimate output;

storing the 3D camera pose estimate output, and providing the previous 3D camera pose estimate;

providing tracked 3D observations based on the 2D image data, the candidate 2D key points, the camera data, the previous 2D image data and candidate 2D key points, the previous 3D camera pose estimate and 3D scene model data from a 3D scene model data source;

providing as a selected 3D camera pose estimate one of the detected 3D camera pose estimate and the previous 3D camera pose estimate based on the detected camera pose estimate and the previous 3D camera pose estimate; and

providing the 3D camera pose estimate output based on the camera data, the detected 3D observations, the tracked 3D observations and the selected 3D camera pose estimate.

11. A method for generating an image of a second object considered stationary in respect of a first object as viewed by a 2D camera imaging the first object, the method comprising :

(i) in a first step related to an acquisition by the camera of an acquired image number n of the first object, providing a grabbed image frame IF_n number n ;

(ii) in a second step related to an acquisition by the camera of an acquired image number n+1 of the first object, providing a grabbed image frame IFn₊₁number n+1 and using one or more processing units for generating a first new 3D camera pose estimate CP_n number n from the grabbed image number n ;

(iii) in a third step related to an acquisition by the camera of an acquired image number n+2 of the first object, providing a grabbed image frame IF_n+2 number n+2, using the one or more processing units for generating a second new 3D camera pose estimate CP_n+1 number n+1 from the grabbed image frame IF _n+1 and generating a generated image frame GF_n number n of the second object as viewed by the camera a 3D camera post represented by the first 3D camera pose estimate CP_n number n.

12. The method of claim 11, further comprising generating a combined image frame CF_n number n by combining the image of the generated image frame GF_n number n with at least one of the images of the acquired images number n, n+1 or n+2.

13. The method of claim 11, further comprising generating a series of images of the second object for a movie of the second object as viewed by the 2D camera imaging the first object by moving the camera while imaging the first object and repeating the method steps (i), (ii) and (iii) for n=1,2,3,...,k.

14. An apparatus that estimates a 3D camera pose relative to a scene from 2D image data of a 2D image frame provided by a camera, the apparatus comprising:

(a) one or more processing units that implement;

a candidate 2D key points detector that receives camera data from a camera data source and the 2D image data from the data interface, and provides candidate 2D key points;

a detected 3D observations detector that receives pre-recorded scene data and the candidate 2D key points, and provides detected 3D observations;

a detected 3D camera pose estimator that receives the camera data, the detected 3D observations and the candidate 2D key points, and provides a detected 3D camera pose estimate;

a tracked 3D observations detector that receives the 2D image data, the candidate 2D key points, the camera data, a previous 2D image data and candidate 2D key points, the previous 3D camera pose estimate and 3D scene model data from a 3D scene model data source, and provides tracked 3D observations;

a pose estimate selector that receives the detected camera pose estimate and the previous 3D camera pose estimate, and provides as a selected 3D camera pose estimate one of the detected 3D camera pose estimate and the previous 3D camera pose estimate; and

a 3D camera pose estimator that receives the camera data, the detected 3D observations, the tracked 3D observations and the selected 3D camera pose estimate, and provides the 3D camera pose estimate output; and

(b) a memory that includes;

a 2D image data and candidate 2D key points storage that (i) receives the detected 2D candidate key points and the 2D image data, and a control input for receiving a control signal related to a 3D camera pose estimate output, (ii) stores the 2D image data and the candidate 2D key points related to the 3D camera pose estimate output, and (iii) provides the previous 2D image data and the candidate 2D key points related to a previous 3D camera pose estimate output; and

a previous 3D camera pose estimate storage that (i) receives the 3D camera pose estimate output, (ii) stores the 3D camera pose estimate output, and (iii) provides the previous 3D camera pose estimate.