Image capture position and image capture direction estimation device, image capture device, image capture position and image capture direction estimation method and program

• US 9,232,128 B2
• Filed: 03/27/2012
• Issued: 01/05/2016
• Est. Priority Date: 03/28/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Inoshita, Tetsuo
• Primary Examiner(s): Berhan, Ahmed A
• Application Number:
  US14/007,411
• Publication Number:
  US 20140015998A1
• Time in Patent Office: 1,379 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

An image capture position and direction estimation device includes a region determination unit that determines a plurality of regions to be associated between a query image and an image with position and direction, a depth estimation unit that estimates a depth of each region, and an image capture position and direction estimation unit that estimates a direction in the region with a large depth estimated by the depth estimation unit and estimates a position in the region with a small depth estimated by the depth estimation unit.

Claims

1. An image capture position direction estimation device comprising:

an image input device configured to input a query image and a predetermined image including position direction information, the query image and the predetermined image having been taken by an image capture device;

a region determination unit that determines a plurality of regions to be associated between the query image and the predetermined image inputted by the image input device;

a depth estimation unit that estimates depth estimation information corresponding to a depth of each of the plurality of regions of the query image according to a determination result of the region determination unit; and

an image capture position and image capture direction estimation unit that estimates, regarding the query image, an image capture direction of the image capture device according to the region with a large depth and an image capture position of the image capture device according to the region with a small depth based on the depth estimation information estimated by the depth estimation unit.

2. The image capture position direction estimation device according to claim 1,

wherein the predetermined image is an image to which a position and a direction are assigned in advance, and

wherein, based on the depth estimation information, the image capture position and image capture direction estimation unit estimates the image capture direction according to a direction assigned to the predetermined image corresponding to the region with the large depth and estimates the image capture position according to a position assigned to the predetermined image corresponding to the region with the small depth.

3. The image capture position direction estimation device according to claim 1, wherein the image capture position and image capture direction estimation unit

determines, based on a depth of the region associated between an image captured by the image capture device and the predetermined image, whether to estimate the position or to estimate a direction from the region,

estimates, as the image capture position, a position of the predetermined image corresponding to the region of which the position is determined to be estimated, and

estimates, as the image capture direction, a direction of the predetermined image corresponding to the region of which the direction is determined to be estimated.

4. The image capture position direction estimation device according to claim 1, wherein the image capture position and image capture direction estimation unit

determines, based on depths of regions associated between an image captured by the image capture device and a plurality of the predetermined images, whether to estimate a position or to estimate a direction from the regions and performs weighting on the predetermined images according to distances between the associated regions,

estimates, as the image capture position, a position of the predetermined image corresponding to the region of which the position is determined to be estimated by multiplying the position by the weight, and

estimates, as the image capture direction, a direction of the predetermined image corresponding to the region of which the direction is determined to be estimated by multiplying the direction by the weight.

5. The image capture position direction estimation device according to claim 1, further comprising:

a first depth information storage unit that stores depth information corresponding to an image capture scene in advance,

wherein the depth estimation unit estimates the depth estimation information by determining an image capture scene of an image captured by the image capture device and reading depth information corresponding to the determined image capture scene from the first depth information storage unit.

6. The image capture position direction estimation device according to claim 1, wherein the region determination unit

extracts a characteristic from each of an image captured by the image capture device and the predetermined image, and

determines, as the plurality of regions to be associated between the image captured by the image capture device and the predetermined image, image regions from which the characteristic is extracted among image regions of the image captured by the image capture device and the predetermined image.

7. The image capture position direction estimation device according to claim 6, further comprising:

a second depth information storage unit that stores a position estimation contribution distribution and a direction estimation contribution distribution in advance according to the depth estimation information in relation with the depth estimation information; and

an image capture position direction estimation amount storage unit that stores a position movement amount distribution and a direction deviation amount distribution according to a distance and a direction between characteristics in relation with the distance and the direction between characteristics,

wherein the image capture position and direction estimation unit

reads the position estimation contribution distribution and the direction estimation contribution distribution from the second depth information storage unit based on the depth estimation information estimated by the depth estimation unit,

reads the position movement amount distribution and the direction deviation amount distribution from the image capture position direction estimation amount storage unit based on the distance and the direction between characteristics corresponding to the region associated by the region determination unit,

estimates the image capture position based on the read position estimation contribution distribution and the read position movement amount distribution, and

estimates the image capture direction based on the read direction estimation contribution distribution and the read direction deviation amount distribution.

8. An image capture device comprising:

the image capture position direction estimation device according to claim 1.

9. An image capture position direction estimation method comprising:

inputting a query image and a predetermined image including position direction information by use of an image input device, the query image and the predetermined image having been taken by an image capture device;

determining a plurality of regions to be associated between the query image and the predetermined image inputted by the image input device;

calculating depth estimation information corresponding to a depth of each of the plurality of regions of the query image according to a determination result of the determining; and

estimating, regarding the query image, an image capture direction of the image capture device according to the region with a large depth and an image capture position of the image capture device according to the region with a small depth based on the depth estimation information estimated in the calculating of the depth estimation information.

10. A non-transitory computer-readable medium comprising program causing a computer to perform:

determining a plurality of regions to be associated between an image captured by an image capture device and a predetermined image;

calculating depth estimation information corresponding to a depth of each of the plurality of regions; and

estimating an image capture direction of the image capture device according to the region with a large depth and estimating an image capture position of the image capture device according to the region with a small depth based on the depth estimation information estimated in the calculating of the depth estimation information.