Device for measuring position and orientation of imaging apparatus and method therefor

• US 10,643,347 B2
• Filed: 02/23/2017
• Issued: 05/05/2020
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kotake, Daisuke
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Beg, Samah A
• Application Number:
  US15/441,086
• Publication Number:
  US 20170249752A1
• Time in Patent Office: 1,167 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An information processing device includes a holding unit which holds pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image in association with each other, an input unit which inputs the captured image, a derivation unit which derives a position and an orientation of the imaging apparatus when the input image is captured, an addition determination unit which determines whether a new piece of reference data is to be added, a generating unit which, if the addition determination unit determines the new piece of reference data is to be added, generates a new piece of reference data, and an updating unit which adds the generated piece of reference data and updates the pieces of reference data.

Claims

1. An information processing apparatus configured to store, in advance, pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, three-dimensional information of a feature in a space corresponding to the image, and an attribute of the feature, wherein the captured image, the position and the orientation, the three-dimensional information, and the attribute are associated with each other, the information processing apparatus comprising:

one or more processors; and

a memory storing executable instructions, which when executed by the one or more processors, cause the information processing apparatus to;

input an image captured by the imaging apparatus;

derive a position and an orientation of the imaging apparatus capturing the input image on the basis of the pieces of reference data stored and the input image;

determine whether the input image, as a new piece of reference data, is to be added on the basis of the derived position and orientation and the pieces of reference data;

decide a feature point that is to be registered into the new piece of reference data in a case where the determination determines that the new piece of reference data is to be added, wherein the feature point that is to be registered is a feature point according to which converted coordinates obtained by converting three-dimensional coordinates of a feature point in a coordinate system of at least one piece of reference data selected based on the derived position and the derived orientation into three-dimensional coordinates in a coordinate system of the input image are included in the input image and according to which the attribute of the feature of a feature point associated with the selected at least one piece of reference data and the attribute of the feature of the feature point in the input image are similar to each other; and

add the new piece of reference data and update the pieces of reference data.

2. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, cause the information processing apparatus to:

in a case where a position and an orientation of the imaging apparatus in a previous frame have been obtained, determine the position and the orientation in the previous frame as prediction values for a present position and orientation of the imaging apparatus.

3. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, cause the information processing apparatus to:

in a case where a position and an orientation of the imaging apparatus in a previous frame have not been obtained, perform relocalization to obtain prediction values for a present position and orientation of the imaging apparatus.

4. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, cause the information processing apparatus to:

select a predetermined or smaller number of feature points from the selected at least one piece of reference data; and

correct prediction values for a present position and orientation of the imaging apparatus by using the selected feature points.

5. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, further cause the information processing apparatus to:

calculate a distance between the derived position of the imaging apparatus and the position of the imaging apparatus in the selected at least one piece of reference data,

wherein a determination whether the new piece of reference data is to be added or not is determined on the basis of the distance.

6. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, further cause the information processing apparatus to:

in a case where feature points in the selected at least one piece of reference data are projected on the input image, calculate a rate of feature points to be projected on the image,

wherein it is determined whether to generate a piece of reference data or not on the basis of the rate.

7. The information processing apparatus according to claim 1, wherein the executable instructions, when executed by the one or more processors, cause the information processing apparatus to:

sequentially designate feature points in the selected at least one piece of reference data;

calculate three-dimensional coordinates and image coordinates of each of the designated feature points in a current frame in a coordinate system with reference to the imaging apparatus;

determine whether the designated feature point is to be added or not on the basis of the three-dimensional coordinates and the image coordinates; and

add, to the pieces of reference data, the feature point determined to be added.

8. The information processing apparatus according to claim 1,

wherein two or more pieces of reference data are sequentially selected from the pieces of reference data, and

wherein prediction values for a present position and orientation of the imaging apparatus are sequentially corrected by using feature points in the sequentially selected pieces of reference data to obtain the derived position and orientation.

9. The information processing apparatus according to claim 1, wherein the attribute of the feature is included in the piece of reference data, and wherein the executable instructions, when executed by the one or more processors, further cause the information processing apparatus to:

calculate a position of the feature on the input image on the basis of the derived position and orientation and three-dimensional information of the feature in the selected at least one piece of reference data; and

determine a degree of contribution of the feature in the piece of reference data to be generated by comparing the attribute of the feature and an attribute of the position of the feature on the input image,

wherein the deriving derives a position and an orientation of the imaging apparatus by using the pieces of updated reference data and the determined degree of contribution.

10. The information processing apparatus according to claim 9, wherein the attribute of the feature includes a luminance and a color.

11. The information processing apparatus according to claim 9, wherein the attribute of the feature includes a distance with reference to the imaging apparatus.

12. The information processing apparatus according to claim 1,

wherein the attribute is a luminance value of the feature.

13. An information processing method for an information processing apparatus configured to store, in advance, pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, three-dimensional information of a feature in a space corresponding to the image, and an attribute of the feature, wherein the captured image, the position and the orientation, the three-dimensional information, and the attribute are associated with each other, the method comprising:

inputting an image captured by the imaging apparatus;

deriving a position and an orientation of the imaging apparatus capturing the input image on the basis of the pieces of reference data stored and the input image;

determining whether the input image, as a new piece of reference data, is to be added on the basis of the derived position and orientation and the pieces of reference data;

deciding a feature point that is to be registered into the new piece of reference data in a case where it is determined that the new piece of reference data is to be added, wherein the feature point that is to be registered is a feature point according to which converted coordinates obtained by converting three-dimensional coordinates of a feature point in a coordinate system of at least one piece of reference data selected based on the derived position and the derived orientation into three-dimensional coordinates in a coordinate system of the input image are included in the input image and according to which the attribute of the feature of a feature point associated with the selected at least one piece of reference data and the attribute of the feature of the feature point in the input image are similar to each other; and

adding the new piece of reference data and updating the pieces of reference data.

14. A non-transitory computer-readable storage medium storing a program including instructions, which when executed by a computer of an information processing apparatus configured to store, in advance, pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, three-dimensional information of a feature in a space corresponding to the image, and an attribute of the feature, wherein the captured image, the position and the orientation, the three-dimensional information, and the attribute are associated with each other, cause the information processing apparatus to:

input an image captured by the imaging apparatus;

derive a position and an orientation of the imaging apparatus capturing the input image on the basis of the pieces of reference data stored and the input image;

determine whether the input image, as a new piece of reference data, is to be added on the basis of the derived position and orientation and the pieces of reference data;

decide a feature point that is to be registered into the new piece of reference data in a case where the determination determines that the new piece of reference data is to be added, wherein the feature point that is to be registered is a feature point according to which converted coordinates obtained by converting three-dimensional coordinates of a feature point in a coordinate system of at least one piece of reference data selected based on the derived position and the derived orientation into three-dimensional coordinates in a coordinate system of the input image are included in the input image and according to which the attribute of the feature of a feature point associated with the selected at least one piece of reference data and the attribute of the feature of the feature point in the input image are similar to each other; and

add the new piece of reference data and update the pieces of reference data.