Localisation and mapping

• US 10,657,663 B2
• Filed: 06/13/2016
• Issued: 05/19/2020
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Martini, Antonio
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US15/180,370
• Publication Number:
  US 20160292867A1
• Time in Patent Office: 1,436 Days
• Field of Search: 382203
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/5378 for displaying an additional top view, e.g. radar screens or maps using two or more virtual cameras concurrently A63F13/5252
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 30/422 : Technical drawings; Geographical maps
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 13/211 : using temporal multiplexing
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method generates a three-dimensional map of a region from successive images of that region captured from different camera poses. The method captures successive images of the region, detects a gravitational vertical direction in respect of each captured image, detects feature points within the captured images and designates a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image. The method also includes, for a captured image (i) deriving a camera pose from detected feature points in the image; (ii) rotating the gravitational vertical direction to the coordinates of a reference keyframe using the camera poses derived for that image and the reference keyframe; and (iii) comparing the rotated direction with the actual gravitational vertical direction for the reference keyframe to detect a quality measure of that image.

Claims

1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with the three dimensional spatial position, the method comprising:

capturing successive images of the region using a camera;

detecting, by a processor, feature points within the captured images;

designating, by the processor, a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image;

detecting, by the processor, an angular velocity of the camera between a previous image and a current image of the successive images using at least one of (i) one or more accelerometers, or (ii) one or more gyroscopes;

in respect of a newly captured image, detecting, by the processor, a pose of the camera by detecting the position of landmark points in the newly captured image, the detecting step comprising;

integrating the angular velocity of the camera since the previous image was captured to obtain an estimated rotation; and

predicting the positions of the landmark points in the newly captured image from the estimated rotation and the position of the landmark points in the previous image.

2. A method according to claim 1, comprising the step of applying a statistical search to detect the position of the landmark points in the newly captured image based upon the predicted positions of the landmark points.

3. A method according to claim 1, comprising executing a camera pose solver to detect a camera pose from the integrated angular velocity and two or more detected landmark points in the image.

4. A method according to claim 3, in which the camera pose solver estimates multiple candidate camera poses from respective subsets of the landmark points and the integrated angular velocity and selects, as a camera pose, a best candidate camera pose based on a comparison with other landmark points.

5. A method according to claim 4, comprising estimating a translation of the camera since the previous image was captured, in which the camera pose solver estimates a camera pose based on the integrated angular velocity and the estimated translation.

6. A method according to claim 1, in which the previous image is an immediately preceding image.

7. A non-transitory computer readable medium storing computer instructions thereon, the instructions, when executed by a processor, cause the processor to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with the three dimensional spatial position, the method comprising:

capturing successive images of the region using a camera;

detecting feature points within the captured images;

designating a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image;

detecting an angular velocity of the camera between a previous image and a current image of the successive images using at least one of (i) one or more accelerometers, or (ii) one or more gyroscopes;

in respect of a newly captured image, detecting a pose of the camera by detecting the position of landmark points in the newly captured image, the detecting step comprising;

integrating the angular velocity of the camera since the previous image was captured to obtain an estimated rotation; and

predicting the positions of the landmark points in the newly captured image from the estimated rotation and the position of the landmark points in the previous image.

8. An image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with the three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image, wherein the image processing apparatus comprises:

a camera configured to capture successive images of the region using, the camera having an associated angular velocity detector configured to detect an angular velocity of the camera between a previous image and a current image of the successive images using at least one of (i) one or more accelerometers, or (ii) one or more gyroscopes; and

a camera pose detector configured, in respect of a newly captured image, to detect a pose of the camera according to the position of landmark points in the newly captured image, the camera pose detector being configured to integrate an angular velocity of the camera since the previous image was captured to obtain an estimated rotation, and to predict the position of the landmark points in the newly captured image from the estimated rotation and the position of the landmark points in the previous image.

9. An image processing apparatus according to claim 8, wherein the image processing apparatus is configured to apply a statistical search to detect the position of the landmark points in the newly captured image based upon the predicted positions of the landmark points.

10. An image processing apparatus according to claim 8, wherein the image processing apparatus is configured to execute a camera pose solver to detect a camera pose from the integrated angular velocity and two or more detected landmark points in the image.

11. An image processing apparatus according to claim 10, in which the camera pose solver estimates multiple candidate camera poses from respective subsets of the landmark points and the integrated angular velocity and selects, as a camera pose, a best candidate camera pose based on a comparison with other landmark points.

12. An image processing apparatus according to claim 11, wherein the image processing apparatus is further configured to estimating a translation of the camera since the previous image was captured, in which the camera pose solver estimates a camera pose based on the integrated angular velocity and the estimated translation.

13. An image processing apparatus according to claim 8, in which the previous image is an immediately preceding image.

14. A computer games apparatus comprising:

an image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with the three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image, wherein the image processing apparatus comprises;

a camera configured to capture successive images of the region using a camera, the camera having an associated angular velocity detector configured to detect an angular velocity of the camera between a previous image and a current image of the successive images using at least one of (i) one or more accelerometers, or (ii) one or more gyroscopes;

a camera pose detector configured, in respect of a newly captured image, to detect a pose of the camera according to the position of landmark points in the newly captured image, the camera pose detector being configured to integrate an angular velocity of the camera since the previous image was captured to obtain an estimated rotation, and to predict the position of the landmark points in the newly captured image from the estimated rotation and the position of the landmark points in the previous image; and

an image renderer configured to render an image for display by combining a captured image with one or more computer-generated image portions generated with respect to map positions associated with feature points of the captured image.

15. A computer games apparatus according to claim 14, wherein the image processing apparatus is configured to apply a statistical search to detect the position of the landmark points in the newly captured image based upon the predicted positions of the landmark points.

16. A computer games apparatus according to claim 14, wherein the image processing apparatus is configured to execute a camera pose solver to detect a camera pose from the integrated angular velocity and two or more detected landmark points in the image.

17. A computer games apparatus according to claim 16, in which the camera pose solver estimates multiple candidate camera poses from respective subsets of the landmark points and the integrated angular velocity and selects, as a camera pose, a best candidate camera pose based on a comparison with other landmark points.

18. A computer games apparatus according to claim 17, wherein the image processing apparatus is further configured to estimating a translation of the camera since the previous image was captured, in which the camera pose solver estimates a camera pose based on the integrated angular velocity and the estimated translation.

19. A computer games apparatus according to claim 14, in which the previous image is an immediately preceding image.