Method and apparatus for displaying a 2D image data set combined with a 3D rangefinder data set

• US 7,477,360 B2
• Filed: 02/09/2006
• Issued: 01/13/2009
• Est. Priority Date: 02/11/2005
• Status: Expired due to Fees

• Current Assignee: Aurora Operations Incorporated (Aurora Innovation, Inc.)
• Original Assignee: DeltaSphere, Inc.
• Inventors: England, James N., Elgin, Benjamin C., Helser, Aron T., Holloway, Richard L.
• Primary Examiner(s): Tarcza, Thomas H
• Assistant Examiner(s): Ratcliffe, Luke D
• Application Number:
  US11/351,248
• Publication Number:
  US 20060244746A1
• Time in Patent Office: 1,069 Days
• Field of Search: 356 401- 41, 356 301- 315, 356 501- 515, 356 6- 22
• US Class Current: 356/4.01
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

Method, computer program product, and apparatus for displaying a 2D image data set combined with a 3D range data set. A 3D range data set and at least one 2D image data set which differs from the 3D range data set in resolution, origin, or both, are provided. The 3D range data set is represented as a first displayed image while each 2D image data set is represented as at least a second displayed image. Corresponding features are identified within each displayed image. 3D transformation is computed between the 3D range data set and at least one 2D image data set using the specified corresponding features to establish geometric correspondence. At least one 2D image data set is represented as a third displayed image, wherein a 3D coordinate can be computed for any specified pixel of each 2D image data set using the 3D transformation and 3D range data set.

Claims

1. A method for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

providing a 3D range data set;

providing at least one 2D image data set having a different resolution than the 3D range data set;

representing the 3D range data set as a first displayed image;

representing the at least one 2D image data set as at least a second displayed image;

identifying corresponding features within the first displayed image and within the at least one second displayed image respectively;

computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on geometry of the identified corresponding features, such that geometric correspondence between the 3D range data set and the at least one 2D image data set is determined;

representing the at least one 2D image data set as a third displayed image; and

computing a 3D coordinate for a specified pixel of the third displayed image based on the previously computed 3D transformation and the 3D range data set.

2. The method of claim 1, wherein the step of representing the at least one 2D image data set as the third displayed image comprises representing at least part of the 3D range data as 3D surfaces and representing at least part of the at least one 2D image data set as texture maps on at least part of the 3D surfaces according to the previously computed 3D transformation.

4. The method of claim 1, further comprising the steps of identifying at least a first feature within the third displayed image;

identifying at least a second feature within one of the first displayed image, the second displayed image, and the third displayed image;

computing a 3D coordinate for at least the first feature based on the at least one 2D image data set, the previously computed 3D transformation, and the 3D range data set; and

computing 3D coordinates for the at least second feature.

5. The method of claim 4, wherein identifying features comprises one of using a computer cursor controlled by a mouse to identify, using a computer cursor controlled by a pointing stick to identify, using a computer cursor controlled by a joystick to identify, using a computer cursor controlled by a touch pad to identify, using software to identify, and using combinations of the above to identify.

6. The method of claim 4, wherein identifying features comprises at least one technique selected from the group consisting of using sub-pixel interpolation in any 2D displayed image wherein a software tool may allow the user to estimate and specify the location of a feature anywhere within a pixel and not just at its origin, using interpolation between measured 3D range data points on surfaces in any 3D displayed image wherein a software tool may allow the user to estimate and specify the location of a feature anywhere upon a surface even if that particular location is not directly associated with a measured 3D range data point, using estimates of the centers of features wherein the user may estimate and specify the location of the center of a feature even if the particular pixel at that chosen center appears no different from adjacent pixels, and using holes and data interpolated across holes wherein the rangefinder device did not acquire a range measurement and wherein a software tool may allow the user to estimate and specify the location of a feature anywhere within a hole even though that particular location is not directly associated with a measured 3D range data point.

7. The method of claim 4, further comprising the step of displaying the computed measurement.

8. The method of claim 7, wherein the step of displaying represents displaying at least one display selected from the group consisting of a measurement point displayed, a plurality of measurement points displayed, a measurement line displayed, a plurality of measurement lines displayed, a measurement surface displayed, a plurality of measurement surfaces displayed, a measurement volume displayed, a plurality of measurement volumes displayed, a measurement feature displayed, a plurality of measurement features displayed, and combinations thereof displayed.

9. The method of claim 1, wherein the at least one 2D image data set is provided by a 2D imaging device selected from the group consisting of a color digital still camera, a monochrome digital still camera, a color digital video camera, a monochrome digital video camera, a color analog video camera with digital conversion, a monochrome analog video camera with digital conversion, a color film camera with the film or print made from the film subsequently being digitized for use as a 2D image data set, a monochrome film camera with the film or print made from the film subsequently being digitized for use as a 2D image data set, a color digital line scan camera, a monochrome digital line scan camera, a color digital panoramic scanning camera, a monochrome digital panoramic scanning camera, another device that acquires a multiplicity of image data points simultaneously, another device that acquires a multiplicity of image data points over a period of time, and combinations of the preceding.

10. The method of claim 1, wherein the 3D range data set is provided by a 3D rangefinder device selected from the group consisting of a scanning laser rangefinder using time of flight range measurement principles, a scanning laser rangefinder using phase comparison range measurement principles, a scanning laser rangefinder using any other range measurement principles, an imaging laser rangefinder range camera using time of flight range measurement principles, an imaging laser rangefinder range camera using phase comparison range measurement principles, an imaging laser rangefinder range camera using any other range measurement principles, a triangulation rangefinder, a stereo image rangefinder, a multiple image rangefinder, any other device that acquires a multiplicity of range data points simultaneously, and any other device that acquires a multiplicity of range data points over a period of time and combinations of the above.

11. The method of claim 1, wherein the displayed image comprises a 2D image selected from the group consisting of a 2D range image comprising range values from the 3D rangefinder device converted to monochrome, a 2D range image comprising range values from the 3D rangefinder device converted to false color, a 2D reflectance image comprising intensity values of the rangefinding signal reflected from a physical surface and thereafter received by the 3D rangefinder device converted to monochrome, a 2D reflectance image comprising intensity values of the rangefinding signal reflected from a physical surface and thereafter received by the 3D rangefinder device converted to false color, a 2D registered color image comprising a color camera image previously registered with 3D range data, a 2D registered color image wherein the image is acquired from the same perspective as the 3D range data set, a 2D registered color image wherein the image is acquired from a different perspective than the 3D range data set, a 2D registered color image wherein the image is acquired with the same resolution as the 3D range data set, a 2D registered color image wherein the image is acquired with different resolution from the 3D range data set, a 2D image displayed in spherical projection format, a 2D image displayed in any other 3D-to-2D projection format, and a 2D registered monochrome image comprising a monochrome camera image previously registered with 3D range data.

12. The method of claim 1, wherein the displayed image comprises a 3D image selected from the group consisting of a 3D point display, a 3D point display in orthogonal projection, a 3D point display in perspective projection, a 3D polygonal mesh, a 3D polygonal mesh in orthogonal projection, a 3D polygonal mesh in perspective projection, a 3D surface geometry display, a 3D surface geometry display in orthogonal projection, and a 3D surface geometry display in perspective projection.

13. The method of claim 1, implemented in a computer processor executing a suitable computer software program product executing therein.

14. The method of claim 1, implemented in a suitable computer software program product embodied on computer readable tangible media.

15. The method of claim 1, wherein the 3D range data set is represented by at least two displayed images.

16. The method of claim 1, further comprising the steps of providing at least a second 3D range data set;

representing the at least second 3D range data set as at least a fourth displayed image;

identifying corresponding features within the fourth displayed image and within the second displayed image respectively; and

computing a 3D transformation between the at least second 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the at least second 3D range data set and the at least one 2D image data set is determined.

17. The method of claim 16, wherein the step of representing the at least one 2D image data set as a third displayed image comprises the step of representing the at least one 2D image data set as a third displayed image;

and wherein a 3D coordinate can be computed for any specified pixel of the at least one 2D image data set based on the previously computed 3D transformation and at least one of the first 3D range data set and the second 3D range data set.

26. The method of claim 1 wherein computing the 3D coordinate for a specified pixel of the third displayed image includes at least one of interpolating between a plurality of 3D range data points and performing sub-pixel interpolation of a plurality of pixels of the third displayed image.

27. The method of claim 1 wherein the 2D image data set has a different origin from the 3D range data set.

3. A method for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

providing a 3D range data set;

providing at least one 2D image data set which differs from the 3D range data set in at least one of resolution, origin, and both resolution and origin;

representing the 3D range data set as a first displayed image;

representing the at least one 2D image data set as at least a second displayed image;

identifying corresponding features within the first displayed image and within at least one second displayed image respectively;

computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the 3D range data set and at least one 2D image data set is determined; and

representing the at least one 2D image data set as a third displayed image wherein a 3D coordinate can be computed for any specified pixel of the at least one 2D image data set based on the previously computed 3D transformation and the 3D range data set, wherein the step of representing the at least one 2D image data set as the third displayed image further comprises the steps of representing the 3D range data set as a fourth displayed image wherein a hot spot area associated with all or some of the at least one 2D image data set is indicated by highlighting therein, selecting the hot spot area interactively, and representing the at least one 2D image data set associated with the selected hot spot area as the third displayed image.

18. A computer program product stored in computer readable media for execution in at least one processor for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

a first software module for providing a first 3D range data set;

a second software module for providing at least one 2D image data set having a different resolution than the 3D range data set;

a third software module for representing the 3D range data set as a first displayed image;

a fourth software module for representing the at least one 2D image data set as at least a second displayed image;

a fifth software module for identifying corresponding features within the first displayed image and within at least one second displayed image respectively;

a sixth software module for computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the 3D range data set and at least one 2D image data set is determined; and

a seventh software module for representing the at least one 2D image data set as a third displayed image and for computing a 3D coordinate for a specified pixel of the third displayed image based on the previously computed 3D transformation and the 3D range data set.

19. The computer program product of claim 18, wherein the step of representing the at least one 2D image data set as the third displayed image comprises representing at least part of the 3D range data as 3D surfaces and representing at least part of the at least one 2D image data set as texture maps on at least part of the 3D surfaces according to the previously computed 3D transformation.

21. The computer program product of claim 18, further comprising the steps of identifying at least a first feature within the third displayed image;

identifying at least a second feature within one of the first displayed image, the second displayed image, and the third displayed image;

computing a 3D coordinate for at least the first feature based on the at least one 2D image data set, the previously computed 3D transformation, and the 3D range data set; and

computing 3D coordinates for the at least second feature.

28. The computer program product of claim 18 wherein computing the 3D coordinate for a specified pixel of the third displayed image includes at least one of interpolating between a plurality of 3D range data points and performing sub-pixel interpolation of a plurality of pixels of the third displayed image.

29. The computer program product of claim 18 wherein the 2D image data set has a different origin from the 3D range data set.

20. A computer program product stored in computer readable media for execution in at least one processor for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

a first software module for providing a first 3D range data set;

a second software module for providing at least one 2D image data set which differs from the 3D range data set in at least one of resolution, origin, and both resolution and origin;

a third software module for representing the 3D range data set as a first displayed image;

a fourth software module for representing the at least one 2D image data set as at least a second displayed image;

a fifth software module for identifying corresponding features within the first displayed image and within at least one second displayed image respectively;

a sixth software module for computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the 3D range data set and at least one 2D image data set is determined; and

a seventh software module for representing the at least one 2D image data set as a third displayed image wherein a 3D coordinate can be computed for any specified pixel of the at least one 2D image data set based on the previously computed 3D transformation and the 3D range data set, wherein the step of representing the at least one 2D image data set as the third displayed image further comprises the steps of representing the 3D range data set as a fourth displayed image wherein a hot spot area associated with all or some of the at least one 2D image data set is indicated by highlighting therein;

selecting the hot spot area interactively; and

representing the at least one 2D image data set associated with the selected hot spot area as the third displayed image.

22. An apparatus for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

at least one computer processor;

a computer program product executing within the at least one computer processor, wherein the computer program product further comprises at least the following software modules therein;

a first software module for providing a first 3D range data set;

a second software module for providing at least one 2D image data set having a different resolution than the 3D range data set;

a third software module for representing the 3D range data set as a first displayed image;

a fourth software module for representing the at least one 2D image data set as at least a second displayed image;

a fifth software module for identifying corresponding features within the first displayed image and within at least one second displayed image respectively;

a sixth software module for computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the 3D range data set and at least one 2D image data set is determined; and

a seventh software module for representing the at least one 2D image data set as a third displayed image and for computing a 3D coordinate for a specified pixel of the third displayed image based on the previously computed 3D transformation and the 3D range data set.

23. The apparatus of claim 22, wherein the step of representing the at least one 2D image data set as the third displayed image comprises representing at least part of the 3D range data as 3D surfaces and representing at least part of the at least one 2D image data set as texture maps on at least part of the 3D surfaces according to the previously computed 3D transformation.

25. The apparatus of claim 22, further comprising the steps of identifying at least a first feature within the third displayed image;

identifying at least a second feature within one of the first displayed image, the second displayed image, and the third displayed image;

computing a 3D coordinate for at least the first feature based on the at least one 2D image data set, the previously computed 3D transformation, and the 3D range data set; and

computing 3D coordinates for the at least second feature.

30. The apparatus of claim 22 wherein computing the 3D coordinate for a specified pixel of the third displayed image includes at least one of interpolating between a plurality of 3D range data points and performing sub-pixel interpolation of a plurality of pixels of the third displayed image.

31. The apparatus of claim 22 wherein the 2D image data set has a different origin from the 3D range data set.

24. An apparatus for displaying a 2D image data set obtained using a 2D imaging device combined with a 3D range data set obtained using a 3D rangefinder device, comprising:

at least one computer processor;

a computer program product executing within the at least one computer processor, wherein the computer program product further comprises at least the following software modules therein;

a first software module for providing a first 3D range data set;

a second software module for providing at least one 2D image data set which differs from the 3D range data set in at least one of resolution, origin, and both resolution and origin;

a third software module for representing the 3D range data set as a first displayed image;

a fourth software module for representing the at least one 2D image data set as at least a second displayed image;

a fifth software module for identifying corresponding features within the first displayed image and within at least one second displayed image respectively;

a sixth software module for computing a 3D transformation between the 3D range data set and the at least one 2D image data set based on the geometry of the specified corresponding features, such that the geometric correspondence between the 3D range data set and at least one 2D image data set is determined; and

a seventh software module for representing the at least one 2D image data set as a third displayed image wherein a 3D coordinate can be computed for any specified pixel of the at least one 2D image data set based on the previously computed 3D transformation and the 3D range data set, wherein the step of representing the at least one 2D image data set as the third displayed image further comprises the steps of representing the 3D range data set as a fourth displayed image wherein a hot spot area associated with all or some of the at least one 2D image data set is indicated by highlighting therein;

selecting the hot spot area interactively; and

representing the at least one 2D image data set associated with the selected hot spot area as the third displayed image.