ESTIMATION DEVICE AND METHOD

• US 20170154429A1
• Filed: 11/16/2016
• Published: 06/01/2017
• Est. Priority Date: 11/27/2015
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: YOSHITAKE, Toshiyuki, MOTEKI, Atsunori, YAMAGUCHI, Nobuyasu
• Granted Patent: US 10,223,804 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method includes associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position and an orientation of a camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time, calculating vectors based on feature descriptors of the first feature points and feature descriptors of the second feature points for respective pairs of the first feature points and the second feature points, determining whether an association of the first feature points and the second feature points is satisfactory, and performing an estimation process that estimates a new position and a new orientation of the camera at the second time when the association is determined to be satisfactory.

Claims

1. A method executed by a processor comprising:

acquiring a first image captured by a camera at a first time;

performing a first process that matches a group of first feature points extracted from the first image with map points having three-dimensional coordinates predefined for an object;

based on a number of map points that match to the first image by the first process, estimating an accuracy of first position and orientation information obtained by using a result of the first process, the first position and orientation information indicating a position and an orientation of a camera at the first time;

storing, in a memory, keyframe information including information of each of the group of first feature points when the estimated accuracy is higher than or equal to a threshold;

acquiring a second image captured by the camera at a second time that is after the first time;

performing the first process on a group of second feature points extracted from the second image and the map points;

based on another number of map points matched to the second image by the first process, estimating an accuracy of second position and orientation information obtained by using another result of the first process, the second position and orientation information indicating a position and an orientation of the camera at the second time;

performing a second process that identifies pairs of respective of the group of second feature points with corresponding first feature points based on the keyframe information to generate a plurality of pairs when the estimated accuracy of the second position and orientation is less than the threshold;

generating vectors for respective of the plurality of pairs;

generating new second position and orientation information by using at least a part of the plurality of pairs when a distribution of the vectors satisfies a predetermined condition; and

outputting the new second position and orientation information to a display control process that performs display control for displaying additional information on the second image based on the new second position and orientation information.

2. The method according to claim 1, wherein

the keyframe information further includes the first position and orientation information, and

the new second position and orientation information is generated by further using the keyframe information when the distribution of the vectors satisfies the predetermined condition.

3. The method according to claim 1, wherein

the first process matches one or more of the first feature points with one or more of the map points based on respective first feature descriptors of the group of first feature points and respective second feature descriptors of the second feature points,

the both of the one or more of the first feature points and the one or more of the map points have feature descriptors greater than another threshold, and

the second process pairs respective ones of the group of second feature points with any of the first feature points in an order from one of the group of second feature points having a third feature descriptor which is most similar to any of the first feature descriptors of the group of first feature points.

4. A method executed by a processor, the method comprising:

associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position of a camera and an orientation of the camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time by the camera;

calculating a plurality of vectors based on feature descriptors of the plurality of first feature points and feature descriptors of the plurality of second feature points for respective pairs of the plurality of first feature points and the plurality of second feature points;

based on a distribution of the vectors, determining whether or not an association of the plurality of first feature points and the plurality of second feature points is satisfactory; and

performing an estimation process that estimates a new position of the camera and a new orientation of the camera at the second time based on pairs of the plurality of first feature points and the plurality of second feature points when the association is determined to be satisfactory.

5. The method according to claim 4, wherein the estimation process at the second time is not performed when the association is determined to be not satisfactory.

6. The method according to claim 4, further comprising:

when an average of lengths of the plurality of vectors is smaller than a first threshold, determining that the association is satisfactory.

7. The method according to claim 4, further comprising:

when a standard deviation of the lengths of the plurality of vectors is smaller than a second threshold, determining that the association is satisfactory.

8. The method according to claim 4, further comprising:

when an average of lengths of the plurality of vectors is smaller than a first threshold and a standard deviation of the lengths of the plurality of vectors is smaller than a second threshold, determining that the association is satisfactory.

9. The method according to claim 4, further comprising:

calculating a ratio of a number of vectors whose distance between a median and each of the plurality of vectors is greater than or equal to a third threshold, the median being determined based on of the plurality of vectors; and

determining that the association is satisfactory when the ratio is less than a fourth threshold.

10. The method according to claim 9, wherein the estimation process estimates the new position and orientation by using pairs of first feature points and second feature points whose distance between the median and an image associated vector is less than a fifth threshold.

11. The method according to claim 5, further comprising:

causing a display device to display information indicating that the estimation process is not performed when the association is determined to be not satisfactory.

12. A device comprising:

a memory; and

a processor coupled to the memory and configured to;

associate a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position of a camera and an orientation of the camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time by the camera,

calculate a plurality of vectors based on feature descriptors of the plurality of first feature points and feature descriptors of the plurality of second feature points for respective pairs of the plurality of first feature points and the plurality of second feature points,

based on a distribution of the vectors, determine whether or not an association of the plurality of first feature points and the plurality of second feature points is satisfactory, and

perform an estimation process that estimates a new position of the camera and a new orientation of the camera at the second time based on pairs of the plurality of first feature points and the plurality of second feature points when the association is determined to be satisfactory.

13. The device according to claim 12, wherein the estimation process at the second time is not performed when the association is determined to be not satisfactory.

14. The device according to claim 12, wherein the processor is configured to, when an average of lengths of the plurality of vectors is smaller than a first threshold, determine that the association is satisfactory.

15. The device according to claim 12, wherein the processor is configured to, when a standard deviation of the lengths of the plurality of vectors is smaller than a second threshold, determine that the association is satisfactory.

16. The device according to claim 12, wherein the processor is configured to, when an average of lengths of the plurality of vectors is smaller than a first threshold and a standard deviation of the lengths of the plurality of vectors is smaller than a second threshold, determine that the association is satisfactory.

17. The device according to claim 12, wherein the processor is configured to:

calculate a ratio of a number of vectors whose distance between a median and each of the plurality of vectors is greater than or equal to a third threshold, the median being determined based on of the plurality of vectors, and

determine that the association is satisfactory when the ratio is less than a fourth threshold.

18. The device according to claim 17, wherein the estimation process estimates the new position and orientation by using pairs of first feature points and second feature points whose distance between the median and an image associated vector is less than a fifth threshold.

19. The device according to claim 13 wherein the processor is configured to cause a display device to display information indicating that the estimation process is not performed when the association is determined to be not satisfactory.