Localisation and mapping

• US 9,415,310 B2
• Filed: 03/13/2015
• Issued: 08/16/2016
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Martini, Antonio
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/657,346
• Publication Number:
  US 20150186746A1
• Time in Patent Office: 522 Days
• Field of Search: 382/203
• US Class Current: 1/1
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/5378 for displaying an additional top view, e.g. radar screens or maps using two or more virtual cameras concurrently A63F13/5252
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 30/422 : Technical drawings; Geographical maps
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 13/211 : using temporal multiplexing
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method generates a three-dimensional map of a region from successive images of that region captured from different camera poses. The method captures successive images of the region, detects a gravitational vertical direction in respect of each captured image, detects feature points within the captured images and designates a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image. The method also includes, for a captured image (i) deriving a camera pose from detected feature points in the image; (ii) rotating the gravitational vertical direction to the coordinates of a reference keyframe using the camera poses derived for that image and the reference keyframe; and (iii) comparing the rotated direction with the actual gravitational vertical direction for the reference keyframe to detect a quality measure of that image.

Claims

1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing successive images of the region;

detecting, by a processor, a gravitational vertical direction in respect of each captured image;

detecting, by the processor, feature points within the captured images;

designating, by the processor, a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe;

in respect of a given image captured at a relocalisation operation;

(i) deriving, by the processor, a camera pose for the given image from detected feature points in the given image;

(ii) rotating, by the processor, the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and

(iii) comparing, by the processor, the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.

2. A method according to claim 1, in which the reference keyframe is an earliest-captured one of the set of keyframes.

3. A method according to claim 1, comprising:

performing tests to detect whether or not to designate the given image as a keyframe, outcomes of the tests varying according to a detected quality measure for that candidate keyframe.

4. A method according to claim 1, in which the actual gravitational vertical direction associated with the reference keyframe depends upon a gravity direction associated with two or more keyframes.

5. A method according to claim 1, in which, if the rotated gravitational vertical direction with the actual gravitational vertical direction associated with the reference keyframe differ by more than a threshold difference, the deriving step is repeated in respect of a different set of landmark points.

6. A non-transitory computer readable medium storing computer instructions thereon, the instructions, when executed by a processor, cause the processor to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:

capturing successive images of the region;

detecting a gravitational vertical direction in respect of each captured image;

detecting feature points within the captured images;

designating a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe;

in respect of a given image captured at a relocalisation operation;

(i) deriving a camera pose for the given image from detected feature points in the given image;

(ii) rotating the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and

(iii) comparing the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.

7. Image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe, wherein the apparatus comprises:

a camera configured to capture successive images of the region;

a gravity detector configured to detect a gravitational vertical direction in respect of each captured image;

a relocalisation controller configured, in respect of a given image captured at a relocalisation operation, to;

(i) derive a camera pose for the given image from detected landmark points in that keyframe;

(ii) rotate the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and

(iii) compare the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.

8. Computer games apparatus comprising:

an image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe, wherein the image processing apparatus comprises;

a camera configured to capture successive images of the region;

a gravity detector configured to detect a gravitational vertical direction in respect of each captured image;

a relocalisation controller configured, in respect of a given image captured at a relocalisation operation, to;

(i) derive a camera pose for the given image from detected landmark points in that keyframe;

(ii) rotate the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and

(iii) compare the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image; and

an image renderer configured to render an image for display by combining a captured image with one or more computer-generated image portions generated with respect to map positions associated with feature points of the captured image.