DEVICE FOR MEASURING POSITION AND ORIENTATION OF IMAGING APPARATUS AND METHOD THEREFOR

• US 20170249752A1
• Filed: 02/23/2017
• Published: 08/31/2017
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kotake, Daisuke
• Granted Patent: US 10,643,347 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An information processing device includes a holding unit which holds pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image in association with each other, an input unit which inputs the captured image, a derivation unit which derives a position and an orientation of the imaging apparatus when the input image is captured, an addition determination unit which determines whether a new piece of reference data is to be added, a generating unit which, if the addition determination unit determines the new piece of reference data is to be added, generates a new piece of reference data, and an updating unit which adds the generated piece of reference data and updates the pieces of reference data.

Claims

1. An information processing device comprising:

a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated with each other;

an input unit configured to input an image captured by the imaging apparatus;

a derivation unit configured to derive a position and an orientation of the imaging apparatus when the input image is captured on the basis of the pieces of reference data and the input image;

an addition determination unit configured to determine whether a new piece of reference data is to be added on the basis of the derived position and orientation and the pieces of reference data;

a generating unit configured to, in a case where the addition determination unit determines the new piece of reference data is to be added, generate a new piece of reference data on the basis of the input image, the derived position and orientation, and three-dimensional information of a feature in one of the pieces of reference data; and

an updating unit configured to add the generated piece of reference data and update the pieces of reference data.

2. The information processing device according to claim 1,

wherein the derivation unit has

a prediction unit configured to obtain prediction values for a present position and orientation of the imaging apparatus,

a selection unit configured to select at least one piece of reference data from the pieces of reference data on the basis of the prediction values, and

a correction unit configured to correct the prediction values by using a feature point in the selected piece of reference data to obtain the position and orientation to be derived.

3. The information processing device according to claim 2, wherein, in a case where a position and an orientation of the imaging apparatus in a previous frame have been obtained, the prediction unit determines the position and the orientation in the previous frame as the prediction values.

4. The information processing device according to claim 2, wherein, in a case where a position and an orientation of the imaging apparatus in a previous frame have not been obtained, the prediction unit performs relocalization to obtain the prediction values.

5. The information processing device according to claim 2, wherein the correction unit selects a predetermined or smaller number of feature points from the selected piece of reference data and corrects the prediction values by using the selected feature points.

6. The information processing device according to claim 2, further comprising

a distance calculation unit configured to calculate a distance between the derived position of the imaging apparatus and the position of the imaging apparatus in the selected piece of reference data,

wherein the addition determination unit determines whether the new piece of reference data is to be added or not on the basis of the distance.

7. The information processing device according to claim 2, further comprising

a rate calculation unit configured to, in a case where feature points in the selected piece of reference data are projected on the input image, calculate a rate of feature points to be projected on the image,

wherein the addition determination unit determines whether the generating unit generates a piece of reference data or not on the basis of the rate.

8. The information processing device according to claim 2,

wherein the generating unit has

a designation unit configured to sequentially designate feature points in the selected piece of reference data,

a coordinate calculation unit configured to calculate three-dimensional coordinates and image coordinates of each of the designated feature points in a current frame in a coordinate system with reference to the imaging apparatus,

a feature point determination unit configured to determine whether the designated feature point is to be added or not on the basis of the three-dimensional coordinates and the image coordinates, and

an addition unit configured to add, to the pieces of reference data, the feature point determined to be added.

9. The information processing device according to claim 2,

wherein the selection unit sequentially selects two or more pieces of reference data from the pieces of reference data, and

wherein the correction unit sequentially corrects the prediction values by using feature points in the sequentially selected pieces of reference data to obtain the position and orientation to be derived.

10. The information processing device according to claim 2, wherein an attribute of the feature is included in the piece of reference data, the information processing device further comprising:

a position calculation unit configured to calculate a position of the feature on the input image on the basis of the derived position and orientation and three-dimensional information of the feature in the selected piece of reference data; and

a determination unit configured to determine a degree of contribution of the feature in the piece of reference data to be generated by comparing the attribute of the feature and an attribute of the position of the feature on the input image,

wherein the derivation unit derives a position and an orientation of the imaging apparatus by using the pieces of reference data updated by the updating unit and the determined degree of contribution.

11. The information processing device according to claim 10, wherein the attribute of the feature includes a luminance and a color.

12. The information processing device according to claim 10, wherein the attribute of the feature includes a distance with reference to the imaging apparatus.

13. An information processing method for an information processing device including a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated with each other, the method comprising:

inputting an image captured by the imaging apparatus;

deriving a position and an orientation of the imaging apparatus when the input image is captured on the basis of the pieces of reference data and the input image;

determining whether a new piece of reference data is to be added on the basis of the derived position and orientation and the pieces of reference data;

generating, in a case where it is determined that the new piece of reference data is to be added, a new piece of reference data on the basis of the input image, the derived position and orientation and three-dimensional information of a feature in one of the pieces of reference data; and

adding the generated piece of reference data and updating the pieces of reference data.

14. A non-transitory computer-readable storage medium storing a program causing a computer to function as:

a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated with each other;

an input unit configured to input an image captured by the imaging apparatus;

a derivation unit configured to derive a position and an orientation of the imaging apparatus when the input image is captured on the basis of the pieces of reference data and the input image;

an addition determination unit configured to determine whether a new piece of reference data is to be added on the basis of the derived position and orientation and the pieces of reference data;

a generating unit configured to, in a case where the addition determination unit determines the new piece of reference data is to be added, generate a new piece of reference data on the basis of the input image, the derived position and orientation, and three-dimensional information of a feature in one of the pieces of reference data; and

an updating unit configured to add the generated piece of reference data and update the pieces of reference data.