Localisation and mapping

• US 9,824,450 B2
• Filed: 07/25/2013
• Issued: 11/21/2017
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Martini, Antonio
• Primary Examiner(s): WILLIAMS, JEFFERY A
• Application Number:
  US14/417,935
• Publication Number:
  US 20150172626A1
• Time in Patent Office: 1,580 Days
• Field of Search: 348 50
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/5378 for displaying an additional top view, e.g. radar screens or maps using two or more virtual cameras concurrently A63F13/5252
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 30/422 : Technical drawings; Geographical maps
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 13/211 : using temporal multiplexing
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method generates a three-dimensional map of a region from successive images of the region captured from different camera poses. The method detects feature points within the captured images and designates a subset of the images as a set of keyframes each having camera pose data and respective sets of measurement data representing image positions of landmark points detected as feature points in that image. The method also includes performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes. And for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, the method generates a non-bundle-adjusted point corresponding to that feature point and derives a camera pose.

Claims

1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing, by an image processing device, successive images of the region;

detecting, by a processor, feature points within the captured images;

designating, by the processor, a subset of the captured images as a set of keyframes each having camera pose data indicative of a camera pose and respective sets of measurement data representing image positions of landmark points detected as feature points in that image;

in respect of the keyframes, the processor performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes by;

(i) deriving a difference between the measurement data for landmark points in the keyframes and corresponding image positions obtained from the camera pose data and the three dimensional spatial position of that landmark point; and

(ii) adjusting one or both of the camera pose data for one or more keyframes and the three dimensional spatial position of one or more landmark points so as to reduce the detected difference;

for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, the processor generating a non-bundle-adjusted point corresponding to that feature point; and

deriving, by the processor, a camera pose in respect of an image using the non-bundle-adjusted points and the bundle adjusted landmark points;

in which the step of generating a non-bundle-adjusted point comprises selecting a subset of one or more feature points of the intervening image to become non-bundle-adjusted points, and the selecting step comprises;

detecting the number of landmark points relating to each of a plurality of different image regions of that intervening image; and

selecting a new feature point to be added as a non-bundle-adjusted point according to a weighting applicable to the image region containing that feature point, such that the weighting is lower for an image region having a higher number of landmark points.

2. A method according to claim 1, comprising:

at a next keyframe, performing bundle adjustment with respect to one or more of the non-bundle-adjusted points so as to convert one or more of the non-bundle-adjusted points into bundle-adjusted landmark points.

3. A method according to claim 1, comprising the step of:

applying a position tracking filter to predict an image position corresponding to a non-bundle-adjusted point from image to image.

4. A method according to claim 3, comprising the steps of:

comparing the predicted image position of a non-bundle-adjusted point with a detected image position of an image feature point corresponding to that non-bundle-adjusted point; and

detecting a statistical quality of the non-bundle-adjusted point from the comparison.

5. A method according to claim 4, comprising the step of:

adopting, as non-bundle-adjusted points, those feature points having at least a threshold statistical quality.

6. A method according to claim 1, in which the selecting step comprises:

detecting an image distance between a feature point and the image position of a nearest bundle-adjusted point in the intervening image; and

selecting a feature point as a non-bundle-adjusted point which has at least a threshold image distance from a nearest a bundle-adjusted point in the intervening image.

7. A method according to claim 1, in which the adjusting step comprises:

deriving a statistical model of the three dimensional position of the landmark points and the camera pose; and

discarding one or more landmark points which have below a threshold fit to the statistical model.

8. A method according to claim 1, in which the refining step operates with respect to a predetermined number of keyframes.

9. A method according to claim 1, in which the step of generating a non-bundle-adjusted point corresponding to the feature point comprises:

detecting an initial approximation of the three dimensional position of the feature point by triangulation between two images having different camera poses.

10. A method according to claim 1, in which the camera pose data comprises data defining a three dimensional camera position and a camera orientation.

11. A non-transitory computer-readable recording medium storing computer instructions thereon, the computer instructions, when executed by a computer, causes the computer to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing successive images of the region;

detecting feature points within the captured images;

designating a subset of the captured images as a set of keyframes each having camera pose data indicative of a camera pose and respective sets of measurement data representing image positions of landmark points detected as feature points in that image;

in respect of the keyframes, performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes by;

(i) deriving a difference between the measurement data for landmark points in the keyframes and corresponding image positions obtained from the camera pose data and the three dimensional spatial position of that landmark point; and

(ii) adjusting one or both of the camera pose data for one or more keyframes and the three dimensional spatial position of one or more landmark points so as to reduce the detected difference;

for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, generating a non-bundle-adjusted point corresponding to that feature point; and

deriving a camera pose in respect of an image using the non-bundle-adjusted points and the bundle adjusted landmark points;

in which generating a non-bundle-adjusted point comprises selecting a subset of one or more feature points of the intervening image to become non-bundle-adjusted points, and the selecting comprises;

detecting the number of landmark points relating to each of a plurality of different image regions of that intervening image; and

selecting a new feature point to be added as a non-bundle-adjusted point according to a weighting applicable to the image region containing that feature point, such that the weighting is lower for an image region having a higher number of landmark points.

12. The non-transitory computer-readable recording medium of claim 11, further comprising:

at a next keyframe, performing bundle adjustment with respect to one or more of the non-bundle-adjusted points so as to convert one or more of the non-bundle-adjusted points into bundle-adjusted landmark points.

13. Image processing apparatus operable to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having camera pose data indicative of a camera pose and respective sets of measurement data representing image positions of landmark points detected as feature points in that image, the apparatus comprising:

a bundle adjuster configured in respect of the keyframes to perform bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes by;

(i) deriving a difference between the measurement data for landmark points in the keyframes and corresponding image positions obtained from the camera pose data and the three dimensional spatial position of that landmark point; and

(ii) adjusting one or both of the camera pose data for one or more keyframes and the three dimensional spatial position of one or more landmark points so as to reduce the detected difference;

a point generator operable, in respect of a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, to generate a non-bundle-adjusted point corresponding to that feature point; and

a camera pose detector operable to derive a camera pose in respect of an image using the non-bundle-adjusted points and the bundle adjusted landmark points;

in which the non-bundle-adjusted point is generated by selecting a subset of one or more feature points of the intervening image to become non-bundle-adjusted points, and the selection comprises;

detection of the number of landmark points relating to each of a plurality of different image regions of that intervening image; and

selection of a new feature point to be added as a non-bundle-adjusted point according to a weighting applicable to the image region containing that feature point, such that the weighting is lower for an image region having a higher number of landmark points.

14. Computer games apparatus comprising:

an image processing apparatus according to claim 13; and

an image renderer configured to render an image for display by combining a captured image with one or more computer-generated image portions generated with respect to map positions associated with feature points of the captured image.

15. Apparatus according to claim 14, comprising a camera configured to capture the images.