Determining camera position for a photograph having a displaced center of projection

• US 9,160,979 B1
• Filed: 05/27/2011
• Issued: 10/13/2015
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: Ulmer, John E.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): BEASLEY, DEIRDRE L
• Application Number:
  US13/118,065
• Time in Patent Office: 1,600 Days
• Field of Search: 348/42, 348/135, 382/106
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An image depicting an object has a center that is not on an optical axis of a camera used to capture the image. The image is on an image plane, and the object is in an object space. A method implemented for automatically determining the position of the camera relative to the object includes displaying the image via a user interface, receiving vanishing point information from the user interface, such that the vanishing point data indicates a respective location on the image plane of each of a first vanishing point, a second vanishing point, and a third vanishing point, and automatically determining the position of the camera based on the first vanishing point, the second vanishing point, and the third vanishing point.

Claims

1. A method implemented in a computing device for automatically determining a position of a camera relative to an object depicted in an image and captured using the camera, wherein the object is in an object space, wherein the image is on an image plane, and wherein a center of the image is not on an optical axis of the camera, the method comprising:

causing the image to be displayed via a user interface of the computing device;

receiving vanishing point data from the user interface, wherein the vanishing point data indicates a respective location on the image plane of each of a first vanishing point, a second vanishing point, and a third vanishing point; and

automatically determining the position of the camera based on the first vanishing point, the second vanishing point, and the third vanishing point, wherein determining the position of the camera includes determining an orthocenter of a vanishing point triangle having vertices including the first vanishing point, the second vanishing point, and the third vanishing point, wherein the orthocenter of the vanishing point triangle is a point formed at an intersection of an altitude corresponding to the first vanishing point, an altitude corresponding to the second vanishing point, and an altitude corresponding to the third vanishing point, wherein the altitude corresponding to the first vanishing point is a line segment extending through the first vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the first vanishing point, wherein the altitude corresponding to the second vanishing point is a line segment extending through the second vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the second vanishing point, and wherein the altitude corresponding to the third vanishing point is a line segment extending through the third vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the third vanishing point, and wherein the optical axis of the camera intersects the image plane at the orthocenter of the vanishing point triangle.

2. The method of claim 1, wherein determining the position of the camera further includes:

determining a distance H_COP from the orthocenter of the vanishing point triangle to a line segment bounded by the first vanishing point and the second vanishing point, wherein a line perpendicular to the line segment that passes through the orthocenter of the vanishing point triangle divides the line segment into two sections of respective lengths D₁ and D₂ at a point P_SEG on the line segment;

determining a length H of a hypotenuse of a right triangle formed by the orthocenter of the vanishing point triangle, the point P_SEG, and the camera, wherein a line segment bounded by the point P_SEG and the camera defines the hypotenuse, wherein the length H=√

(D₁D₂); and

determining a height H_EYE of the right triangle based on the distance H_COP and the length H.

3. The method of claim 1, wherein a projection of the object onto the image plane conforms to a pinhole camera model.

4. The method of claim 1, wherein the image is a cropped version of an original image, wherein a center of the original image is on the optical axis of the camera.

5. The method of claim 1, wherein at least a portion of the object is represented using mutually orthogonal lines.

6. The method of claim 1, wherein receiving vanishing point data includes:

causing a first set of moveable pins to be displayed over the image, wherein the first set of moveable pins includes a first pair of pins to specify a first line on the image plane and a second pair of pins to specify a second line on the image plane;

causing a second set of moveable pins to be displayed over the image, wherein the second set of moveable pins includes a third pair of pins to specify a third line on the image plane and a fourth pair of pins to specify a fourth line on the image plane; and

causing a third set of moveable pins to be displayed over the image, wherein the third set of moveable pins includes a fifth pair of pins to specify a fifth line on the image plane and a sixth pair of pins to specify a sixth line on the image plane; and

in response to a user positioning each of the first set of moveable pins, the second set of moveable pins, and the third set of moveable pins;

automatically determining the first vanishing point, wherein the first vanishing point is at an intersection of the first line and the second line;

automatically determining the second vanishing point, wherein the first vanishing point is at an intersection of the third line and the fourth line; and

automatically determining the third vanishing point, wherein the third vanishing point is at an intersection of the fifth line and the sixth line.

7. A method implemented in a computing device, the method comprising:

receiving first data indicative of a location of a first vanishing point on an image plane, wherein the image plane includes an image of an object;

receiving second data indicative of a location of a second vanishing point on the image plane;

receiving third data indicative of a location of a third vanishing point on the image plane;

automatically determining an orthocenter of a vanishing point triangle having vertices that include the first vanishing point, the second vanishing point, and the third vanishing point, wherein the orthocenter of the vanishing point triangle is a point where an altitude corresponding to the first vanishing point, an altitude corresponding to the second vanishing point, and an altitude corresponding to the third vanishing point intersect, wherein the altitude corresponding to the first vanishing point is a line segment extending through the first vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the first vanishing point, wherein the altitude corresponding to the second vanishing point is a line segment extending through the second vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the second vanishing point, and wherein the altitude corresponding to the third vanishing point is a line segment extending through the third vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the third vanishing point;

automatically determining a position of a camera used to capture the image, wherein the camera is disposed on an optical axis orthogonal to the image plane and intersecting the image plane at the orthocenter of the vanishing point triangle; and

determining a distance between the camera and the orthocenter of the vanishing point triangle along the optical axis.

8. The method of claim 7, wherein determining the distance between the camera and the orthocenter of the vanishing point triangle includes:

determining a distance H_COP from the orthocenter of the vanishing point triangle to a line

segment bounded by the first vanishing point and the second vanishing point, wherein a line perpendicular to the line segment that passes through the orthocenter of the vanishing point triangle divides the line segment into two sections of respective lengths D₁ and D₂ at a point P_SEG on the line segment;

determining a length H of a hypotenuse of a right triangle formed by the orthocenter of the vanishing point triangle, the point P_SEG, and the camera, wherein a line segment bounded by the point PSEG and the camera defines the hypotenuse, wherein the length H=√

(D₁D₂); and

determining a height H_EYE of the right triangle based on the distance H_COP and the length H, wherein the height H_EYE corresponds to the distance between the camera and the orthocenter of the vanishing point triangle.

9. The method of claim 7, wherein the object is in a three-dimensional object space, and wherein receiving each of the first data, the second data, and the third data includes:

receiving respective data descriptive of a projection of a respective pair of parallel lines in the object space onto the image plane, wherein the projection defines a respective pair of nonparallel lines in the image plane; and

determining a respective intersection of the respective pair of non-parallel lines, wherein the intersection corresponds to a respective one of the first vanishing point, the second vanishing point, and the third vanishing point.

10. The method of claim 7, wherein a center of the image does not coincide with the orthocenter of the vanishing point triangle.

11. The method of claim 10, wherein the image is a cropped version of an original image, wherein a center of the original image coincides with the orthocenter of the vanishing point triangle.

12. The method of claim 7, wherein at least a portion of the object is represented using mutually orthogonal lines.

19. The non-transitory computer-readable medium of claim 12, wherein to receive vanishing the point data from the user interface, the instructions further cause the one or more processors to:

cause a first set of moveable pins to be displayed over the image, wherein the first set of moveable pins includes a first pair of pins to specify a first line on the image plane and a second pair of pins to specify a second line on the image plane;

cause a second set of moveable pins to be displayed over the image, wherein the second set of moveable pins includes a third pair of pins to specify a third line on the image plane and a fourth pair of pins to specify a fourth line on the image plane; and

cause a third set of moveable pins to be displayed over the image, wherein the third set of moveable pins includes a fifth pair of pins to specify a fifth line on the image plane and a sixth pair of pins to specify a sixth line on the image plane; and

in response to a user positioning the first set of moveable pins, the second set of moveable pins, and the third set of moveable pins;

automatically determine the first vanishing point, wherein the first vanishing point is at an intersection of the first line and the second line;

automatically determine the second vanishing point, wherein the first vanishing point is at an intersection of the third line and the fourth line; and

automatically determine the third vanishing point, wherein the third vanishing point is at an intersection of the fifth line and the sixth line.

13. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to:

cause an image depicting an object to be displayed via a user interface of a computing device, wherein the object is in an object space, wherein the image is on an image plane, and wherein a center of the image is not on an optical axis of a camera used to capture the image;

receive vanishing point data from the user interface, wherein the vanishing point data indicates a respective location on the image plane of each of a first vanishing point, a second vanishing point, and a third vanishing point; and

determine a position of the camera relative to the object depicted in the image based on the first vanishing point, the second vanishing point, and the third vanishing point, wherein the position of the camera is based on an orthocenter of a vanishing point triangle having vertices that include the first vanishing point, the second vanishing point, and the third vanishing point, wherein the orthocenter of the vanishing point triangle is a point where an altitude corresponding to the first vanishing point, an altitude corresponding to the second vanishing point, and an altitude corresponding to the third vanishing point intersect, wherein the altitude corresponding to the first vanishing point is a line segment extending through the first vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the first vanishing point, wherein the altitude corresponding to the second vanishing point is a line segment extending through the second vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the second vanishing point, and wherein the altitude corresponding to the third vanishing point is a line segment extending through the third vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the third vanishing point.

14. The non-transitory computer-readable medium of claim 13, wherein the optical axis of the camera intersects the image plane at the orthocenter of the vanishing point triangle.

15. The non-transitory computer-readable medium of claim 14, wherein to determining the position of the camera, the instructions further cause the one or more processors to:

determine a distance H_COP from the orthocenter of the vanishing point triangle to a line segment bounded by the first vanishing point and the second vanishing point, wherein a line perpendicular to the line segment that passes through the orthocenter of the vanishing point

triangle divides the line segment into two sections of respective lengths D₁ and D₂ at a point P_SEG on the line segment;

determine a length H of a hypotenuse of a right triangle formed by the orthocenter of the vanishing point triangle, the point P_SEG, and the camera, wherein a line segment bounded by the point P_SEG and the camera defines the hypotenuse, wherein the length H=√

(D₁D₂); and

determine a height H_EYE of the right triangle based on the distance H_COP and the length H.

16. The non-transitory computer-readable medium of claim 13, wherein a projection of the object onto the image plane conforms to a pinhole camera model.

17. The non-transitory computer-readable medium of claim 13, wherein the image is a cropped version of an original image, wherein a center of the original image is on the optical axis of the camera.

18. The non-transitory computer-readable medium of claim 13, wherein at least a portion of the object is represented using mutually orthogonal lines.

20. A method implemented in a computing device for automatically determining a position of a camera relative to an object depicted in an image, wherein the object is in an object space, wherein the image is on an image plane, the method comprising:

causing the image to be displayed via a user interface of the computing device;

determining a first vanishing point from a first pair of lines displayed on the image plane, wherein the first vanishing point corresponds to an intersection of the first pair of lines on the image plane, wherein the first pair of lines correspond to a first set of moveable pins displayed over the image, wherein the first set of moveable pins includes a first pair of pins to specify a first line on the image plane and a second pair of pins to specify a second line on the image plane, and wherein the first pair of lines comprises the first line and the second line;

determining a second vanishing point from a second pair of lines displayed on the image plane, wherein the second vanishing point corresponds to an intersection of the second pair of lines on the image plane;

determining a third vanishing point from a third pair of lines displayed on the image plane, wherein the third vanishing point corresponds to an intersection of the third pair of lines on the image plane;

determining vanishing point data, wherein the vanishing point data indicates a respective location on the image plane of each of the first vanishing point, the second vanishing point, and the third vanishing point; and

automatically determining the position of the camera based on the first vanishing point, the second vanishing point, and the third vanishing point.

21. The method of claim 20, wherein the first pair of lines are parallel lines in the object space.

22. The method of claim 20, wherein determining the position of the camera includes determining an orthocenter of a vanishing point triangle having vertices including the first vanishing point, the second vanishing point, and the third vanishing point, wherein the orthocenter of the vanishing point triangle is a point formed at an intersection of an altitude corresponding to the first vanishing point, an altitude corresponding to the second vanishing point, and an altitude corresponding to the third vanishing point, wherein the altitude corresponding to the first vanishing point is a line segment extending through the first vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the first vanishing point, wherein the altitude corresponding to the second vanishing point is a line segment extending through the second vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the second vanishing point, and wherein the altitude corresponding to the third vanishing point is a line segment extending through the third vanishing point and perpendicular to a side of the vanishing point triangle that is opposite to the third vanishing point.