Augmented image display using a camera and a position and orientation sensor

• US 9,367,962 B2
• Filed: 10/09/2014
• Issued: 06/14/2016
• Est. Priority Date: 10/22/2013
• Status: Active Grant

• Current Assignee: Topcon Positioning Systems Incorporated (Topcon Corporation)
• Original Assignee: Topcon Positioning Systems Incorporated (Topcon Corporation)
• Inventors: Di Federico, Ivan Giovanni, Reytlinger, Sergey, Lanovets, Vyacheslav, Hallett, Jason
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US14/510,444
• Publication Number:
  US 20150109509A1
• Time in Patent Office: 614 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 1/04 combined with cameras
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An augmented image is generated by capturing a visual image of a site with a digital camera, generating a virtual image or associated information from a digital model of the site, and superimposing the virtual image or associated information on the visual image. To register the digital model with the visual image, a sensor pole is introduced into the field of view, and a combined visual image of the site and an optical target on the sensor pole is captured. The position and orientation of the sensor pole with respect to the site reference frame are measured by sensors mounted on the sensor pole; the position and orientation of the digital camera with respect to the sensor pole are calculated from image analysis of the optical target on the sensor pole; and the position and orientation of the digital camera with respect to the site reference frame are calculated.

Claims

1. A method for generating an augmented image, the method comprising the steps of:

receiving combined visual image data representing a combined visual image captured with a digital camera, wherein the combined visual image data comprises;

site visual image data representing a visual image of a site; and

optical target visual image data representing a visual image of at least a portion of an optical target disposed on a reference surface of a position and orientation sensor unit;

identifying, in the combined visual image data, the optical target visual image data;

calculating, based at least in part on the optical target visual image data, a position and an orientation of the digital camera relative to the position and orientation sensor unit;

receiving measurements of a position and an orientation of the position and orientation sensor unit relative to a site reference frame;

calculating, based at least in part on the position and the orientation of the digital camera relative to the position and orientation sensor unit and based at least in part on the position and the orientation of the position and orientation sensor unit relative to the site reference frame, a position and an orientation of the digital camera relative to the site reference frame;

generating, based at least in part on the position and the orientation of the digital camera relative to the site reference frame and based at least in part on a digital model of the site, at least one graphical representation selected from the group consisting of;

a virtual image associated with the site;

information associated with the site; and

information associated with the virtual image; and

displaying on a display the at least one graphical representation superimposed on a displayed visual image selected from the group consisting of;

a displayed site visual image displaying at least a portion of the visual image of the site; and

a displayed combined visual image displaying at least a portion of the visual image of the site and displaying at least a portion of the visual image of at least a portion of the optical target.

2. The method of claim 1, further comprising the steps of:

disposing the digital camera such that the site is within a field of view of the digital camera;

disposing the position and orientation sensor unit such that the at least a portion of the optical target is within the field of view of the digital camera; and

capturing the combined visual image with the digital camera.

3. The method of claim 1, further comprising the step of:

displaying on the display the displayed combined visual image, wherein the step of displaying on the display the displayed combined visual image is performed after the step of receiving combined visual image data and before the step of identifying, in the combined visual image data, the optical target visual image data.

4. The method of claim 1, wherein the position and orientation sensor unit comprises at least one sensor and at least one sensor measurement system operably coupled to the at least one sensor, further comprising the step of:

measuring, with the at least one sensor and with the at least one sensor measurement system, the position and the orientation of the position and orientation sensor unit relative to the site reference frame.

5. The method of claim 4, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one global navigation satellite system antenna and at least one global navigation satellite system receiver operably coupled to the at least one global navigation satellite system antenna; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

6. The method of claim 4, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one optical prism and at least one total station operably coupled to the at least one optical prism; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

7. The method of claim 1, wherein the digital camera comprises a lens and an image sensor, further comprising the steps of:

receiving calibration parameters comprising;

a focal length of the lens; and

a display scaling factor representing a ratio between an image height on the display and an image height on the image sensor; and

receiving a digital model of the optical target;

wherein;

the step of identifying, in the combined visual image data, the optical target visual image data is based at least in part on the calibration parameters and is based at least in part on the digital model of the optical target;

the step of calculating, based at least in part on the optical target visual image data, a position and an orientation of the digital camera relative to the position and orientation sensor unit is based at least in part on the calibration parameters and is based at least in part on the digital model of the optical target; and

the step of generating, based at least in part on the position and the orientation of the digital camera relative to the site reference frame and based at least in part on a digital model of the site, at least one graphical representation selected from the group consisting of;

a virtual image associated with the site;

information associated with the site; and

information associated with the virtual image;

is further based at least in part on the calibration parameters.

8. A system for generating an augmented image, the system comprising:

a digital camera;

a sensor pole comprising;

a sensor head comprising at least one sensor;

a shaft coupled to the sensor head; and

an optical target disposed on a reference surface of the shaft;

at least one sensor measurement system operably coupled to the at least one sensor;

a display; and

a computational system operably coupled to the digital camera, the at least one sensor measurement system, and the display;

wherein;

when the digital camera and the sensor pole are disposed such that a site and at least a portion of the optical target are within a field of view of the digital camera, the digital camera is configured to;

capture a combined visual image, wherein the combined visual image comprises;

a visual image of the site; and

a visual image of the at least a portion of the optical target;

represent the combined visual image as combined visual image data, wherein the combined visual image data comprises;

site visual image data representing the visual image of the site; and

optical target visual image data representing the visual image of the at least a portion of the optical target; and

transmit the combined visual image data;

the at least one sensor and the at least one sensor measurement system are configured to;

measure a position and an orientation of the sensor pole relative to a site reference frame;

represent the position and the orientation of the sensor pole as measurement data; and

transmit the measurement data; and

the computational system is configured to;

receive the combined visual image data;

receive the measurement data;

identify, in the combined visual image data, the optical target visual image data;

calculate, based at least in part on the optical target visual image data, a position and an orientation of the digital camera relative to the sensor pole;

calculate, based at least in part on the position and the orientation of the digital camera relative to the sensor pole and based at least in part on the position and the orientation of the sensor pole relative to the site reference frame, a position and an orientation of the digital camera relative to the site reference frame;

generate, based at least in part on the position and the orientation of the digital camera relative to the site reference frame and based at least in part on a digital model of the site, at least one graphical representation selected from the group consisting of;

a virtual image associated with the site;

information associated with the site; and

information associated with the virtual image; and

display the at least one graphical representation superimposed on a displayed visual image selected from the group consisting of;

a displayed site visual image displaying at least a portion of the visual image of the site; and

a displayed combined visual image displaying at least a portion of the visual image of the site and displaying at least a portion of the visual image of at least a portion of the optical target.

9. The system of claim 8, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one global navigation satellite system antenna and at least one global navigation satellite system receiver operably coupled to the at least one global navigation satellite system antenna; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

10. The system of claim 8, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one optical prism and at least one total station operably coupled to the at least one optical prism; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

11. The system of claim 8, wherein:

the digital camera comprises a lens and an image sensor; and

the computational system is further configured to;

receive calibration parameters comprising;

a focal length of the lens; and

a display scaling factor representing a ratio between an image height on the display and an image height on the image sensor;

receive a digital model of the optical target;

identify, in the combined visual image data, the optical target visual image data based at least in part on the calibration parameters and based at least in part on the digital model of the optical target;

calculate the position and the orientation of the digital camera relative to the site reference frame further based at least in part on the calibration parameters and further based at least in part on the digital model of the optical target; and

generate the at least one graphical representation further based at least in part on the calibration parameters.

12. The system of claim 8, wherein the digital camera, the display, and the computational system are integrated into a laptop computer or a tablet computer.

13. The system of claim 8, wherein:

the digital camera, the display, and the computational system are integrated into a head-mounted unit; and

the display is a heads-up display.

14. The system of claim 8, wherein:

the digital camera is mounted on an eyeglass frame;

the display is a heads-up display mounted on the eyeglass frame; and

the computational system comprises a first computational system and a second computational system, wherein;

the first computational system is mounted on the eyeglass frame;

the second computational system is not mounted on the eyeglass frame;

the first computational system and the second computational system are operably coupled; and

the first computational system is operably coupled to the digital camera and to the display.

15. The system of claim 8, wherein:

the shaft is a survey pole having a cylindrical surface; and

the optical target is disposed on at least a portion of the cylindrical surface.

16. An apparatus for generating an augmented image, the apparatus comprising:

at least one processor unit;

memory operably coupled to the at least one processor unit;

a display operably coupled to the at least one processor unit; and

at least one data storage device operably coupled to the at least one processor unit, wherein the at least one data storage device stores computer program instructions which, when executed by the at least one processor unit, causes the at least one processor unit to perform the method comprising the steps of;

receiving combined visual image data representing a combined visual image captured with a digital camera, wherein the combined visual image data comprises;

site visual image data representing a visual image of a site; and

optical target visual image data representing a visual image of at least a portion of an optical target disposed on a reference surface of a position and orientation sensor unit;

identifying, in the combined visual image data, the optical target visual image data;

calculating, based at least in part on the optical target visual image data, a position and an orientation of the digital camera relative to the position and orientation sensor unit;

receiving measurements of a position and an orientation of the position and orientation sensor unit relative to a site reference frame;

calculating, based at least in part on the position and the orientation of the digital camera relative to the position and orientation sensor unit and based at least in part on the position and the orientation of the position and orientation sensor unit relative to the site reference frame, a position and an orientation of the digital camera relative to the site reference frame;

generating, based at least in part on the position and the orientation of the digital camera relative to the site reference frame and based at least in part on a digital model of the site, at least one graphical representation selected from the group consisting of;

a virtual image associated with the site;

information associated with the site; and

information associated with the virtual image; and

displaying on the display the at least one graphical representation superimposed on a displayed visual image selected from the group consisting of;

a displayed site visual image displaying at least a portion of the visual image of the site; and

a displayed combined visual image displaying at least a portion of the visual image of the site and displaying at least a portion of the visual image of the at least a portion of the optical target.

17. The apparatus of claim 16, wherein the method further comprises the step of:

displaying on the display the displayed combined visual image, wherein the step of displaying on the display the displayed combined visual image is performed after the step of receiving combined visual image data and before the step of identifying, in the combined visual image data, the optical target visual image data.

18. The apparatus of claim 16, wherein the position and orientation sensor unit comprises at least one sensor and at least one sensor measurement system operably coupled to the at least one sensor.

19. The apparatus of claim 18, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one global navigation satellite system antenna and at least one global navigation satellite system receiver operably coupled to the at least one global navigation satellite system antenna; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

20. The apparatus of claim 18, wherein the at least one sensor and the at least one sensor measurement system operably coupled to the at least one sensor comprises:

at least one optical prism and at least one total station operably coupled to the at least one optical prism; and

at least one sensor and at least one measurement system operably coupled to the at least one sensor selected from the group consisting of;

an electronic magnetic compass; and

an inertial measurement unit.

21. The apparatus of claim 16, wherein the digital camera comprises a lens and an image sensor, and wherein the method further comprises the steps of:

receiving calibration parameters comprising;

a focal length of the lens; and

a display scaling factor representing a ratio between an image height on the display and an image height on the image sensor; and

receiving a digital model of the optical target;

wherein;

the step of identifying, in the combined visual image data, the optical target visual image data is based at least in part on the calibration parameters and is based at least in part on the digital model of the optical target;

the step of calculating, based at least in part on the optical target visual image data, a position and an orientation of the digital camera relative to the position and orientation sensor unit is based at least in part on the calibration parameters and is based at least in part on the digital model of the optical target; and

the step of generating, based at least in part on the position and the orientation of the digital camera relative to the site reference frame and based at least in part on a digital model of the site, at least one graphical representation selected from the group consisting of;

a virtual image associated with the site;

information associated with the site; and

information associated with the virtual image;

is further based at least in part on the calibration parameters.