Localisation and mapping

• US 9,704,244 B2
• Filed: 03/13/2015
• Issued: 07/11/2017
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Martini, Antonio
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US14/657,546
• Publication Number:
  US 20150187133A1
• Time in Patent Office: 851 Days
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/5378 for displaying an additional top view, e.g. radar screens or maps using two or more virtual cameras concurrently A63F13/5252
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 30/422 : Technical drawings; Geographical maps
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 13/211 : using temporal multiplexing
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method generates a three-dimensional map of a region from successive images captured from different camera poses. The method includes a camera capturing images of the region; designating a subset of captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image; for a newly captured image, generating a camera pose prediction; selecting an ordering of the landmark points; detecting whether an image region of the new image approximately matches an expected image position of a landmark point based on the predicted camera pose by detecting whether the image region is substantially identical to the image information associated with that landmark point; and refining the pose prediction from the detecting step. The selecting includes performing a statistical test on the landmark points and selecting the ordering according to the statistical test results.

Claims

1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing successive images of the region using a camera;

designating, by one or more processors, a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image;

in respect of a newly captured image, detecting, by the one or more processors, a position of the camera by detecting the position of landmark points in the newly captured image, the detecting step comprising;

generating, by the one or more processors, a prediction of the camera pose;

dividing the newly captured image into a plurality of image regions;

selecting, by the one or more processors, an ordering of the landmark points separately in respect of each of the image regions so as to select a second ordering of landmark points for each image region; and

detecting, by the one or more processors, whether an image region of the newly captured image approximately matches an expected image position of a landmark point based on the predicted camera pose by detecting, for successive landmark points in the ordering, whether the image region is substantially identical to the image information associated with that landmark point; and

refining, by the one or more processors, the prediction of the camera pose based on the detecting step;

in which the selecting step comprises;

performing, by the one or more processors, a statistical test on the set of landmark points; and

selecting, by the one or more processors, the ordering according to the results of the statistical test.

2. A method according to claim 1, in which the performing step comprises detecting those landmark points which have achieved at least a threshold proportion of matches over a predetermined number of recent operations of the detecting step in respect of those landmark points.

3. A method according to claim 1, in which the performing step comprises detecting landmark points having spaced-apart predicted positions in the newly captured image.

4. A method according to claim 1, comprising carrying out the detecting step in respect of an image region until a required number of detections has been achieved for that image region.

5. A method according to claim 1, in which, if fewer than a required number of landmark points are identified in respect of an image region, the difference between the actual number and the required number is redistributed amongst other image regions as an increase in their respective required numbers.

6. A method according to claim 1, in which the step of predicting the camera pose comprises deriving a camera pose for the newly captured image from a camera pose associated with a previous image and a detected angular velocity of the camera between the previous image and the current image.

7. A method according to claim 1, in which the selecting step comprises selecting a random ordering of the landmark points, such that the position of a landmark point in the random ordering is partly dependent on the results of the statistical test.

8. A non-transitory computer readable medium storing computer instructions thereon, the instructions, when executed by a processor, cause the processor to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing successive images of the region using a camera;

designating a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image;

in respect of a newly captured image, detecting a position of the camera by detecting the position of landmark points in the newly captured image, the detecting step comprising;

generating a prediction of the camera pose;

dividing the newly captured image into a plurality of image regions;

selecting an ordering of the landmark points separately in respect of each of the image regions so as to select a second ordering of landmark points for each image region; and

detecting whether an image region of the newly captured image approximately matches an expected image position of a landmark point based on the predicted camera pose by detecting, for successive landmark points in the ordering, whether the image region is substantially identical to the image information associated with that landmark point; and

refining the prediction of the camera pose based on the detecting step;

in which the selecting step comprises;

performing a statistical test on the set of landmark points; and

selecting the ordering according to the results of the statistical test.

9. Image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image, the apparatus comprising:

a predicted position detector configured in respect of a newly captured image to detect a prediction of the camera pose for the newly captured image;

a divider configured to divide the newly captured image into a plurality of image regions;

a selector configured to select an ordering of the landmark points separately in respect of each of the image regions so as to select a second ordering of landmark points for each image region;

a detector configured to detect whether an image region of the newly captured image approximately matches an expected image position of a landmark point based on the predicted camera pose by detecting, for successive landmark points in the ordering, whether the image region is substantially identical to the image information associated with that landmark point; and

a prediction processor configured to refine the prediction of the camera pose based on the detection;

in which the selector comprises;

processing means for performing a statistical test on the set of landmark points; and

processing means for selecting the ordering according to the results of the statistical test.

10. Computer games apparatus comprising:

an image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image, the apparatus comprising;

a predicted position detector configured in respect of a newly captured image to detect a prediction of the camera pose for the newly captured image;

a divider configured to divide the newly captured image into a plurality of image regions;

a selector configured to select an ordering of the landmark points separately in respect of each of the image regions so as to select a second ordering of landmark points for each image region;

a detector configured to detect whether an image region of the newly captured image approximately matches an expected image position of a landmark point based on the predicted camera pose by detecting, for successive landmark points in the ordering, whether the image region is substantially identical to the image information associated with that landmark point; and

a prediction processor configured to refine the prediction of the camera pose based on the detection;

in which the selector comprises;

processing means for performing a statistical test on the set of landmark points; and

processing means for selecting the ordering according to the results of the statistical test; and

an image renderer configured to render an image for display by combining a captured image with one or more computer-generated image portions generated with respect to map positions associated with feature points of the captured image.

11. Apparatus according to claim 10, comprising a camera configured to capture the images.