Method, apparatus for shape estimation, and nontransitory computerreadable storage medium

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
0Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. A method for shape estimation, performed by a computer, the method comprising:
 executing extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference;
executing estimation processing that includesfirst processing for selecting first feature points from the plurality of feature points such that a component determined as a noise is removed from the selected first feature points,second processing for estimating a threedimensional shape of the subject based on the selected first feature points after the first processing is performed,third processing for obtaining second feature points by removing the selected first feature points from the plurality of feature points after the estimation of the threedimensional shape is performed by the second processing with the selected first feature points, andfourth processing for executing the first processing, the second processing, and the third processing on the second feature points as the plurality of feature points from which the feature points are removed by the third processing; and
executing integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing,wherein the third processing is configured todivide a region including the selected first feature points into a plurality of partial regions when the estimation of the threedimensional shape of the subject is performed successfully; and
obtain the second feature points by removing, from the plurality of partial regions, a partial region where a ratio of included feature points other than the selected first feature points is smaller than a threshold.
1 Assignment
0 Petitions
Accused Products
Abstract
A method for shape estimation includes: executing extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference; executing estimation processing that includes first processing for selecting feature points from the plurality of feature points; second processing for estimating a threedimensional shape of the subject based on the selected feature points; third processing for removing the feature points from the plurality of feature points after the estimation of the threedimensional shape based on the feature points; and fourth processing for executing the first processing, the second processing, and the third processing on the plurality of feature points from which the feature points are removed by the third processing; and executing integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing.
11 Citations
No References
METHOD FOR PROCESSING A VIDEO DATA SET  
Patent #
US 20100158482A1
Filed 05/02/2008

Current Assignee
IMCUBE LABS GMBH

Sponsoring Entity
IMCUBE LABS GMBH

System and related methods for automatically aligning 2D images of a scene to a 3D model of the scene  
Patent #
US 20080310757A1
Filed 06/11/2008

Current Assignee
BRAINSTORM TECHNOLOGY LLC

Sponsoring Entity
BRAINSTORM TECHNOLOGY LLC

Method for recovering 3D scene structure and camera motion directly from image intensities  
Patent #
US 6,999,612 B1
Filed 08/31/2000

Current Assignee
NEC Corporation

Sponsoring Entity
NEC Laboratories America Inc

Method for recovering 3D scene structure and camera motion from points, lines and/or directly from the image intensities  
Patent #
US 6,859,549 B1
Filed 09/01/2000

Current Assignee
NEC Corporation

Sponsoring Entity
NEC Laboratories America Inc

GENERATING THREEDIMENSIONAL MODELS FROM IMAGES  
Patent #
US 20120041722A1
Filed 02/03/2010

Current Assignee
Hong Kong University of Science and Technology

Sponsoring Entity
Hong Kong University of Science and Technology

COMBINING IMAGES CAPTURED WITH DIFFERENT COLOR PATTERNS  
Patent #
US 20120106840A1
Filed 10/28/2010

Current Assignee
Apple Inc.

Sponsoring Entity
Apple Inc.

METHOD, APPARATUS AND SYSTEM FOR DETECTING A SUPPORTING SURFACE REGION IN AN IMAGE  
Patent #
US 20140119604A1
Filed 10/28/2013

Current Assignee
Canon Kabushiki Kaisha

Sponsoring Entity
Canon Kabushiki Kaisha

Method and System for Recovery of 3D Scene Structure and Camera Motion From a Video Sequence  
Patent #
US 20140147032A1
Filed 11/11/2011

Current Assignee
The Provost Fellows Foundation Scholars and other members of Board of the College of the Holy and Undivided Trinity of Queen Elizabeth near Dublin

Sponsoring Entity
The Provost Fellows Foundation Scholars and other members of Board of the College of the Holy and Undivided Trinity of Queen Elizabeth near Dublin

METHOD, SYSTEM AND APPARATUS FOR DETERMINING A PROPERTY OF AN IMAGE  
Patent #
US 20150063684A1
Filed 08/27/2014

Current Assignee
Canon Kabushiki Kaisha

Sponsoring Entity
Canon Kabushiki Kaisha

Geoaccurate threedimensional reconstruction via imagebased geometry  
Patent #
US 9,466,143 B1
Filed 05/03/2013

Current Assignee
Harris Corporation

Sponsoring Entity
Harris Corporation

Stereoscopic Imaging Using Mobile Computing Devices Having FrontFacing And RearFacing Cameras  
Patent #
US 20170324941A1
Filed 04/26/2017

Current Assignee
InsideMaps Inc.

Sponsoring Entity
InsideMaps Inc.

6 Claims
 1. A method for shape estimation, performed by a computer, the method comprising:
executing extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference; executing estimation processing that includes first processing for selecting first feature points from the plurality of feature points such that a component determined as a noise is removed from the selected first feature points, second processing for estimating a threedimensional shape of the subject based on the selected first feature points after the first processing is performed, third processing for obtaining second feature points by removing the selected first feature points from the plurality of feature points after the estimation of the threedimensional shape is performed by the second processing with the selected first feature points, and fourth processing for executing the first processing, the second processing, and the third processing on the second feature points as the plurality of feature points from which the feature points are removed by the third processing; and executing integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing, wherein the third processing is configured to divide a region including the selected first feature points into a plurality of partial regions when the estimation of the threedimensional shape of the subject is performed successfully; and obtain the second feature points by removing, from the plurality of partial regions, a partial region where a ratio of included feature points other than the selected first feature points is smaller than a threshold.  View Dependent Claims (2)
 3. An apparatus for shape estimation, the apparatus comprising:
a memory; and a processor coupled to the memory and configured to execute extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference; execute estimation processing that includes first processing for selecting first feature points from the plurality of feature points such that a component determined as a noise is removed from the selected first feature points, second processing for estimating a threedimensional shape of the subject based on the selected first feature points after the first processing is performed, third processing for obtaining second feature points by removing the selected first feature points from the plurality of feature points after the estimation of the threedimensional shape is performed by the second processing with the selected first feature points, and fourth processing for executing the first processing, the second processing, and the third processing on the second feature points as the plurality of feature points from which the feature points are removed by the third processing; and execute integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing, wherein the third processing is configured to divide a region including the selected first feature points into a plurality of partial regions when the estimation of the threedimensional shape of the subject is performed successfully; and obtain the second feature points by removing, from the plurality of partial regions, a partial region where a ratio of included feature points other than the selected first feature points is smaller than a threshold.  View Dependent Claims (4)
 5. A nontransitory computerreadable storage medium for storing a program for shape estimation, the program causing a computer to execute a process, the process comprising:
executing extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference; executing estimation processing that includes first processing for selecting first feature points from the plurality of feature points such that a component determined as a noise is removed from the selected first feature points, second processing for estimating a threedimensional shape of the subject based on the selected first feature points after the first processing is performed, third processing for obtaining second feature points by removing the selected first feature points from the plurality of feature points after the estimation of the threedimensional shape is performed by the second processing with the selected first feature points, and fourth processing for executing the first processing, the second processing, and the third processing on the second feature points as the plurality of feature points from which the feature points are removed by the third processing; and executing integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing, wherein the third processing is configured to divide a region including the selected first feature points into a plurality of partial regions when the estimation of the threedimensional shape of the subject is performed successfully; and obtain the second feature points by removing, from the plurality of partial regions, a partial region where a ratio of included feature points other than the selected first feature points is smaller than a threshold.  View Dependent Claims (6)
1 Specification
This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2016183497, filed on Sep. 20, 2016, the entire contents of which are incorporated herein by reference.
The embodiment discussed herein is related to a method, an apparatus for shape estimation, and a nontransitory computerreadable storage medium.
There is a technique for reconstructing a threedimensional shape of a subject based on image information of the subject photographed by a camera from multiple points of view. For example, the threedimensional shape of the subject is information in which multiple feature points forming the subject on a three dimension and threedimensional coordinates are associated.
A conventional technique called structure from motion (SfM) is used to reconstruct the threedimensional shape from multiple pieces of image information photographed by one camera with a point of view changed
The procedure 1 is explained with reference to
The SfM calculates feature points of each of the pieces of image information 10 to 12 based on scaleinvariant feature transform (SIFT) feature values. In
The procedure 2 is explained with reference to
The SfM associates the feature points and the projection points in each of the pieces of image information and executes a search while changing values of threedimensional coordinates and camera parameters of the map points such that a square sum of differences between the associated feature points and projection points is minimized. In an example illustrated in
In the SfM, if noise is included in image information, the threedimensional coordinates and camera parameters of map points deviating from the optimum values are sometimes estimated because of the noise.
Examples of the related art include Japanese Laidopen Patent Publication Nos. 2000194859, 2002032745, 2012208759, and 2014063376.
According to an aspect of the invention, a method for shape estimation includes: executing extraction processing that includes extracting a plurality of feature points from a plurality of pieces of image information of a same subject photographed with a time difference; executing estimation processing that includes first processing for selecting feature points from the plurality of feature points; second processing for estimating a threedimensional shape of the subject based on the selected feature points; third processing for removing the feature points from the plurality of feature points after the estimation of the threedimensional shape based on the feature points; and fourth processing for executing the first processing, the second processing, and the third processing on the plurality of feature points from which the feature points are removed by the third processing; and executing integration processing that includes integrating a plurality of the threedimensional shapes estimated by the estimating processing.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.
In the related art, when the subject is a moving body, it is difficult to estimate a threedimensional shape of the subject from multiple pieces of image information photographed by one camera.
The SfM is a technique based on the premise that the subject stands still. Therefore, if the subject moves, the subject may be determined as noise and removed by the RANSAC. As a result, in the SfM in the related art, it is difficult to estimate a threedimensional shape of a subject which does not stand still. On the other hand, without performing noise removal by the RANSAC as explained with reference to
Image information 18 is image information photographed by the camera 50 at time t1. Image information 19 is image information photographed by the camera 50 at time t2. Note that the position of the camera 50 at time t1 and the position of the camera 50 at time t2 are different. A subject 17a and a subject 17b are present in the image information 18 and 19. The subject 17a stands still and the subject 17b is moving. In this case, feature points of the subject 17b are regarded as noise and are removed during corresponding point fitting. Therefore, it is possible to estimate only a threedimensional shape of the subject 17a out of the subjects 17a and 17b.
Note that, when the subject is a moving body, it is possible to estimate a threedimensional shape using multiple pieces of image information photographed by multiple cameras that synchronize with one another. However, control is complicated and development cost increases.
In one aspect, an object of the present disclosure is to provide a technique for shape estimation that may estimate a threedimensional shape of a subject from multiple pieces of image information photographed by one camera.
An embodiment of a shape estimating method, a shape estimating apparatus, and a shape estimating program disclosed by this application is explained in detail below with reference to the drawings. Note that the present disclosure is not limited by the embodiment.
Before the explanation of this embodiment, camera parameters are explained and the SfM and the RANSAC are explained in detail.
The camera parameters are explained. The camera parameters include external parameters and internal parameters. The external parameters include a translation vector T representing the position of a camera and a rotation matrix R representing the direction of the camera. The translation vector T is indicated by Expression (1) and the rotation matrix R is indicated by Expression (2).
The internal parameters include a camera matrix A representing a focal length and the like of the camera and a distortion parameter representing distortion of a lens. The camera matrix A is indicated by Expression (3). In Expression (3), fx and fy indicate focal lengths and cx and cy indicate principal points (center coordinates) of image information photographed by the camera. The distortion parameter is indicated by n components.
Conversion of a coordinate system performed using the camera parameters is explained.
Subsequently, the threedimensional coordinate system is converted to satisfy a condition A1 and a condition A2 described below.
Condition A1: The position of the camera is the origin (0, 0, 0).
Condition A2: The direction of the camera is a positive direction of a z axis and a projection surface 41 is on z=1.
A conversion formula for converting the threedimensional coordinate system to satisfy the condition A1 and the condition A2 is Expression (7). When coordinates converted by Expression (7) are projected on the projection surface 41 on z=1, converted coordinates are (x/z, y/z, 1). In the following explanation, the coordinates (x/z, y/z, 1) are replaced as indicated by Expression (8).
A conversion formula for converting the coordinates (x, y) of the projection surface 41 into coordinates of an image plane 40 of the camera is Expression (9).
In view of the above explanation, a relation between the coordinates (X, Y, Z) of the point P on the threedimensional coordinates illustrated in
The SfM is explained in detail.
n: The number of feature points (map points) on a threedimensional space
qj: Threedimensional coordinates (estimation target) of a jth feature point (map point) 1≤j≤n
m: The number of pieces of image information
zij: Twodimensional coordinates (known) of the jth feature point in ith image information 45 1≤i≤m
Ai: A 3×3 camera matrix (an estimation target) of a camera that photographs the ith image information 45
Ri: A 3×3 translation rotation matrix (an estimation target) of the camera that photographs the ith image information 45
Ti: A 3×1 translation component (an estimation target) of the camera that photographs the ith image information 45
pi: arrangement of unknown parameters of Ai, Ri, and Ti
The twodimensional coordinates of a projection point obtained by projecting a map point on the image information 45 are defined by Expression (13). In Expression (13), z′ij(pi, qj) may be represented by Expression (14).
A reprojection error is defined. A square error between the twodimensional coordinates zij of the feature point of the image information 45 and the projection point z′ij(pi, qj) is defined by Expression (15).
[z_{ij}−z′_{ij}(p_{i},q_{j})]^{2} (15)
An error obtained by adding up the square error defined by Expression (15) in all pieces of image information i and all map points j is defined as “reprojection error”. The reprojection error is represented by Expression (16).
The SfM iterates calculation of the reprojection error defined by Expression (16) while changing a value of the camera parameter pi and calculates values of the threedimensional coordinates qi and the camera parameter pi with which the reprojection error is minimized. The calculated threedimensional coordinates qi are a threedimensional shape of the subject.
The RANSAC is explained in detail. The RANSAC executes a procedure (a), a procedure (b), a procedure (c), a procedure (d), and a procedure (e) in order as explained below.
The procedure (a) is explained. The RANSAC selects, at random, k feature points out of n feature points included in the image information 45. Note that variables are defined as described below.
K: A set of feature point indexes selected from the image information 45
A known condition is as described below.
 zij: Twodimensional coordinates of a jth (j∈K) feature point in ith (1≤i≤m) image information.
Estimation targets are as described below.
 qj: Threedimensional coordinates (j∈K) of the jth feature point (map point)
 pi: A camera parameter (1≤i≤m)
The procedure (b) is explained. The RANSAC calculates a parameters pi, qj for minimizing Expression (17).
The procedure (c) is explained. After calculating the parameter pi, qj, the RANSAC checks the distance between zij and z′ij(pi, qi) for all zij and calculates an evaluation function indicating how close the distance is. In the following explanation, a calculation expression of the evaluation function is explained.
The evaluation function is represented as F(x). However, x of the evaluation function F(x) is defined by Expression (18). A local evaluation value f indicated by Expression (19) is defined. A threshold th is a value set in advance. Note that the local evaluation value f may be a normal distribution indicated by Expression (20). σ is a value set in advance.
The RANSAC calculates an evaluation function F(x) according to Expression (21).
The procedure (d) is explained. The RANSAC registers a value of the evaluation function F(x) calculated in the procedure (c) and a parameter x respectively in variables F* and x*.
The procedure (e) is explained. The RANSAC iterates the calculations of the procedure (a) to the procedure (c) and, when F(x)>F* in the respective repetitions, updates F* and x* in processing of the procedure (d).
The variable x* calculated by the procedure (a) to the procedure (e) is as indicated by Expression (22).
x*=(p_{1}*, . . . , p_{m}*, q_{1}*, . . . , q_{n}*) (22)
A set Q defined by Expression (23) among threedimensional positions q1*, . . . , and qn* in Expression (22) above is set as feature points (map points), which are not noise, to reconstruct a threedimensional shape. A condition described on the right side of Expression (23) may be, for example, a condition such as “a ratio of the reprojection error equal to or lower than a threshold th2 in an ith image information is 80 percent or more”.
A calculation example of the RANSAC is explained.
Conditions in performing the calculation by the RANSAC are represented as conditions B1, B2, and B3 described below.
Condition B1: Points present near the straight line of y=2x+5 are 200 points. The points present near the straight line are indicated by diamond marks. The points present near the straight line shift according to a normal distribution of a standard deviation 0.1 from the straight line.
Condition B2: Points arranged at random are 200 points. The points arranged at random are indicated by square marks. The points are arranged at random in a range of 0≤x≤10 and 5≤y≤25.
Condition B3: A straight line is estimated by the RANSAC in a state in which the 400 points in total are not distinguished.
Procedures of the calculation examples of the RANSAC are represented as a procedure C1, a procedure C2, a procedure C3, and a procedure C4.
Procedure C1: Five points are selected at random.
Procedure C2: The five points are regressionanalyzed to calculate a and b.
Procedure C3: An absolute value difference in y coordinate are calculated between the estimated straight line (y=ax+b) and each of the input 400 points. The number of points with the absolute value differences equal to or smaller than 0.3 among the 400 points is counted as the number of votes.
Procedure C4: The procedures C1 to C3 are iterated to update, at any time, parameters with a large number of votes as optimum values in a present state.
An example of first calculation by the RANSAC is explained with reference to
An example of second calculation by the RANSAC is explained with reference to
An example of third calculation by the RANSAC is explained with reference to
An example of fourth calculation by the RANSAC is explained with reference to
An example of fifth calculation by the RANSAC is explained with reference to
As explained above, for example, by performing the calculation by the RANSAC, the optimum values “a=2.037, b=4.886, and the number of votes 195” are estimated.
Processing by the shape estimating apparatus according to this embodiment is explained.
Processing of the procedure X1 of the shape estimating apparatus is explained with reference to
The image information 60 is information on an image photographed by the camera 50 at time t1. The image information 61 is information on an image photographed by the camera 50 at time t2. The position of the camera at time t1 and the position of the camera at time t2 are different. The pieces of image information 60 and 61 include a subject 60a and a subject 60b. The subject 60a is a stationary object and the subject 60b is a moving body.
After executing the procedure by the RANSAC and removing noise included in the image information 60 and 61, the shape estimating apparatus executes the SfM to estimate a threedimensional shape 70a of the subject 60a. Note that, since the subject 60b is the moving body, the subject 60b is regarded as noise and removed by the RANSAC.
Processing of the procedure X2 of the shape estimating apparatus is explained with reference to
The shape estimating apparatus projects the threedimensional shape 70a on the image information 60 and specifies and removes a region of the subject 60a corresponding to the threedimensional shape 70a to generate image information 601. The shape estimating apparatus projects the threedimensional shape 70a on the image information 61 and specifies and removes the region of the subject 60a corresponding to the threedimensional shape 70a to generate image information 611.
Processing of the procedure X3 of the shape estimating apparatus is explained with reference to
Processing of the procedure X4 of the shape estimating apparatus is explained. The shape estimating apparatus projects the threedimensional shape 70b on the image information 601 and specifies and removes a region of the image information 601 corresponding to the threedimensional shape 70b to generate image information 602. The shape estimating apparatus projects the threedimensional shape 70b on the image information 611 and specifies and removes the region of the image information 611 corresponding to the threedimensional shape 70b to generate image information 612.
Processing of the procedure X5 of the shape estimating apparatus is explained with reference to
The procedure X6 of the shape estimating apparatus is explained with reference to
Y1 in
When the shape estimating apparatus executes first SfM to estimate the threedimensional shape 70a, the position of the camera 50 (time t1) is represented as a position 50a(1) and the position of the camera 50 (time t2) is represented as a position 50b(1). When the shape estimating apparatus executes second SfM to estimate the threedimensional shape 70b, the position of the camera 50 (time t1) is represented as a position 50a(2) and the position of the camera 50 (time t2) is represented as a position 50b(2). When the shape estimating apparatus executes third SfM to estimate the threedimensional shape 70c, the position of the camera 50 (time t1) is represented as a position 50a(3) and the position of the camera 50 (time t2) is represented as a position 50b(3).
Relative positions of the position 50a(1) and the position 50b(1) are represented as relative positions (1). Relative positions of the position 50a(2) and the position 50b(2) are represented as relative positions (2). Relative positions of the position 50a(3) and the position 50b(3) are represented as relative positions (3). Then, the relative positions (1), the relative positions (2), and the relative positions (3) are different.
The shape estimating apparatus performs coordinate conversion of the threedimensional shapes 70a to 70c to change the relative positions (1), the relative positions (2), and the relative positions (3) to be the same and thereafter performs integration. In an example illustrated in
The configuration of the shape estimating apparatus according to this embodiment is explained.
The input unit 110 is an input interface that receives input of image information photographed by the camera 50 at different times. The input unit 110 outputs multiple pieces of image information to the control unit 130. Information on the photographing times is given to the image information received by the input unit 110.
The storing unit 120 includes an image table 121, a threedimensional shape table 122, and an integrated shape information 123. The storing unit 120 corresponds to a semiconductor memory device such as a random access memory (RAM), a read only memory (ROM), or a flash memory or a storage device such as a hard disk drive (HDD).
The image table 121 is a table that retains information on image information.
The threedimensional shape table 122 includes information on a threedimensional shape estimated by executing the SfM.
The integrated shape information 123 is information obtained as a result of integrating the threedimensional shapes of the threedimensional shape table 122.
Referring back to
The receiving unit 131 is a processing unit that receives multiple pieces of image information from the input unit 110. The receiving unit 131 stores photographing time and image information in the image table 121 in association with each other. The receiving unit 131 iterates the processing every time image information is received from the input unit 110.
The extracting unit 132 is a processing unit that extracts feature points from the respective pieces of image information stored in the image table 121 and stores information on the extracted feature points in the image table 121 in association with the image information. For example, the extracting unit 132 calculates feature points and feature values corresponding to the feature points based on an SIFT feature value of the image information.
The estimating unit 133 is a processing unit that iterates removal of noise by the RANSAC and estimation of a threedimensional shape by the SfM based on the image information photographed at the photographing times and the information on the feature points to estimate a threedimensional shape at every stage. For example, processing by the estimating unit 133 corresponds to the processing of the procedures X1 to X5 explained above.
The estimating unit 133 executes the processing of the procedure X1 illustrated in
The estimating unit 133 executes the procedure X3 illustrated in
The estimating unit 133 executes the procedure X5 illustrated in
An example of processing in which the estimating unit 133 deletes a region of image information corresponding to a successfully estimated threedimensional shape is explained.
Referring to
Referring to
Referring to
Parameters of the camera estimated by kth repetition processing (SfM) and positions of threedimensional shapes are defined as described below.
Ai(k): A 3×3 camera matrix of the camera that photographs ith image information.
Ri(k): A 3×3 rotation matrix of the camera that photographs the ith image information.
Ti(k): A 3×1 translation component of the camera that photographs the ith image information
qj(k): Threedimensional coordinates of a jth feature point (map point)
Referring to
Coordinates of map points of the threedimensional shape 70b estimated by the second SfM are represented as qj(2). Parameters, which are estimated by the second SfM, of the camera 50 that photographs an image at time t1 are represented as A1(2), R1(2), and T1(2). Parameters, which are estimated by the second SfM, of the camera 50 that photographs an image at time t2 are represented as A2(2), R2(2), and T2(2).
Coordinates of map points of the threedimensional shape 70c estimated by the third SfM are represented as qj(3). Parameters, which are estimated by the third SfM, of the camera 50 that photographs an image at time t1 are represented as A1(3), R1(3), and T1(3). Parameters, which are estimated by the third SfM, of the camera 50 that photographs an image at time t2 are represented as A2(3), R2(3), and T2(3).
Referring to
First processing for performing coordinate conversion of qj(1) of the threedimensional shape is explained. Since values of the camera parameters A2(1), R2(1), and T2(1) do not change, the integrating unit 134 keeps qj(1) as it is.
Second processing for converting qj(2) of the threedimensional shape is explained. The camera parameters A2(2), R2(2), and T2(2) are standardized as “A2(1), R2(1), and T2(1)”. Therefore, the integrating unit 134 converts the coordinates qj(2) of map points of the threedimensional shape 70b into qj(2)′ according to processing for converting A2(2), R2(2), and T2(2) into “A2(1), R2(1), and T2(1)”.
Third processing for converting qj(3) of the threedimensional shape is explained. The camera parameters A2(3), R2(3), and T2(3) are standardized as “A2(1), R2(1), and T2(1)”. Therefore, the integrating unit 134 converts coordinates qj(3) of map points of the threedimensional shape 70c into qj(3)′ according to processing for converting A2(3), R2(3), and T2(3) into “A2(1), R2(1), and T2(1)”.
After performing the coordinate conversion of the map points of the threedimensional shapes 70b and 70c, the integrating unit 134 integrates the threedimensional shapes 70a, 70b, and 70c to obtain the threedimensional shape 80b after the integration. The integrating unit 134 causes the storing unit 120 to store information on the threedimensional shape 80b as the integrated shape information 123.
An example of coordinate conversion executed by the integrating unit 134 is explained.
Since the left side of Expression (24) and the left side of Expression (25) are equal, Expression (26) holds. Since the camera itself is the same, Ai(1) may be set equal to Ai(k). Consequently, Expression (26) changes to Expression (27). When Expression (27) is arranged, the threedimensional coordinates (X′, Y′, Z′) after the conversion may be represented by Expression (28).
According to a result of Expression (28), when the camera parameters Ai(k), Ri(k), and Ti(k) corresponding to the threedimensional coordinates qj(k) of the map points are converted into Ai(1), Ri(1), and Ti(1), threedimensional coordinates qj(k)′ after the conversion may be represented by Expression (29). Ai(k) is set equal to Ai(1).
q_{j}^{(k)′}=R_{i}^{(1)}^{−1}[R_{i}^{(k)}q_{i}^{(k)}+T_{i}^{(k)}−T_{i}^{(1)}] (29)
The integrating unit 134 calculates qj(2)′ and qj(3)′ based on Expression (29). The integrating unit 134 combines qj(1), qj(2)′, and qj(3)′ to reconstruct the threedimensional shape 80b.
A processing procedure of the shape estimating apparatus 100 according to this embodiment is explained.
The estimating unit 133 of the shape estimating apparatus 100 determines whether a specific condition is satisfied (step S102). The specific condition is a condition that a ratio of an area of a region not deleted in image information is equal to or smaller than a threshold. Alternatively, the specific condition is a condition that the number of feature points extracted from the region not deleted in the image information is equal to or smaller than a threshold. The shape estimating apparatus 100 may use either one of the specific conditions or may use both of the specific conditions.
When the specific condition is not satisfied (No in step S102), the estimating unit 133 shifts to step S103. When the specific condition is satisfied (Yes in step S102), the estimating unit 133 shifts to step S105.
Step S103 is explained. The estimating unit 133 executes the SfM and the RANSAC to estimate the “parameter pi of the camera” and the “threedimensional coordinates qi of the threedimensional shape of the subject” (step S103). The estimating unit 133 removes, from the pieces of image information, a region where a threedimensional shape may be estimated (step S104) and shifts to step S102.
Step S105 is explained. The integrating unit 134 of the shape estimating apparatus 100 receives designation of reference time in the case of integration (step S105). The integrating unit 134 converts threedimensional coordinates of each of the pieces of threedimensional shape information according to the reference time (step S106). The integrating unit 134 integrates all the pieces of threedimensional shape information (step S107).
An effect of the shape estimating apparatus 100 according to this embodiment is explained. When succeeding in estimation of a threedimensional shape by the SfM based on multiple pieces of image information of the subject photographed at different timings, the shape estimating apparatus 100 iterates processing for estimating a threedimensional shape in the remaining region excluding a region of image information corresponding to the estimated threedimensional shape. The shape estimating apparatus 100 integrates threedimensional shapes estimated at stages to calculate a threedimensional shape corresponding to the subject. Consequently, it is possible to estimate a threedimensional shape of the subject from multiple pieces of image information photographed by one camera. Even if the subject is a moving body, it is possible to estimate a threedimensional shape.
The shape estimating apparatus 100 converts, according to designated parameters of the camera, threedimensional coordinates of the threedimensional shapes estimated at the stages and thereafter performs integration. Consequently, it is possible to combine the threedimensional shapes in a state in which a relative relation among the camera parameters estimated at the stages is standardized.
The shape estimating apparatus 100 divides a region on image information corresponding to a successfully estimated threedimensional shape into multiple partial regions and removes, from the multiple partial regions, a partial region where a ratio of included feature points not corresponding to the threedimensional shape is smaller than a threshold. Consequently, it is possible to appropriately remove, from the pieces of image information, a region not corresponding to the estimated threedimensional shape.
An example of a hardware configuration of a computer that realizes functions same as the functions of the shape estimating apparatus 100 explained in the embodiment is explained.
As illustrated in
The hard disk device 207 includes a reception program 207a, an extraction program 207b, an estimation program 207c, and an integration program 207d. The CPU 201 reads out the reception program 207a, the extraction program 207b, the estimation program 207c, and the integration program 207d and develops the programs in the RAM 206.
The reception program 207a functions as a reception process 206a. The extraction program 207b functions as an extraction process 206b. The estimation program 207c functions as an estimation process 206c. The integration program 207d functions as an integration process 206d.
Processing by the reception process 206a corresponds to the processing by the receiving unit 131. Processing by the extraction process 206b corresponds to the processing by the extracting unit 132. Processing by the estimation process 206c corresponds to the processing by the estimating unit 133. Processing by the integration process 206d corresponds to the processing by the integrating unit 134.
Note that the programs 207a to 207d do not have to be stored in the hard disk device 207 from the beginning. For example, the programs 207a to 207d are stored in a “portable physical medium” such as a flexible disk (FD), a CDROM, a DVD disk, a magnetooptical disk, or an IC card inserted into the computer 200. The computer 200 may read out and execute the programs 207a to 207d.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiment of the present invention has been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.