Determining the relative position between a thermal camera and a 3D camera using a hybrid phantom

• US 10,796,446 B2
• Filed: 09/06/2017
• Issued: 10/06/2020
• Est. Priority Date: 05/23/2017
• Status: Active Grant

• Current Assignee: Brainlab AG
• Original Assignee: Brainlab AG
• Inventors: Rohde, Daniel
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US16/094,002
• Publication Number:
  US 20200273199A1
• Time in Patent Office: 1,126 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

A hybrid phantom including a planar surface that includes a background surface having one of a first property and a second property, and a plurality of marker surfaces. Each marker surface of the plurality of marker surfaces having the other one of the first property and the second property. The first property involves having a high thermal emissivity of 0.8 or higher and the second property involves having a low thermal emissivity of 0.2 or less. One of the first property and the second property further involves being diffuse reflective and the other one of the first property and the second property further involves being specular reflective. A method using a system comprising the hybrid phantom, a first, thermal camera, a second, three-dimensional (3D) camera, and a computer.

Claims

1. A hybrid phantom comprising:

a planar surface comprising;

a background surface having one of a first property and a second property; and

a plurality of marker surfaces, each marker surface of the plurality of marker surfaces having the other one of the first property and the second property,

wherein the first property involves having a high thermal emissivity of 0.8 or higher and the second property involves having a low thermal emissivity of 0.2 or less, and

wherein one of the first property and the second property further involves being diffuse reflective and the other one of the first property and the second property further involves being specular reflective.

20. The hybrid phantom of claim 1, further comprising metallic inlays forming the plurality of marker surfaces.

2. A method using a system comprising a hybrid phantom, a first, thermal camera, a second, three-dimensional (3D) camera, and a computer, the hybrid phantom comprising:

a planar surface comprising;

a background surface having one of a first property and a second property; and

a plurality of marker surfaces, each marker surface of the plurality of marker surfaces having the other one of the first property and the second property,

wherein the first property involves having a high thermal emissivity of 0.8 or higher and the second property involves having a low thermal emissivity of 0.2 or less, and

wherein one of the first property and the second property further involves being diffuse reflective and the other one of the first property and the second property further involves being specular reflective,

the method comprising;

capturing, using the first thermal camera, a thermal image of the hybrid phantom;

identifying, by the computer, an edge of each of the plurality of marker surfaces in the thermal image;

generating, by the computer, thermal marker locations by identifying locations of each of the plurality of marker surfaces from the identified edges of each of the plurality of marker sufaces;

capturing, using the second, 3D camera, a 3D point cloud of the hybrid phantom by emitting light onto the hybrid phantom and analyzing reflected light;

generating, by the computer, 3D marker locations by identifying locations of each of the plurality of marker surfaces from the 3D point cloud captured by the second, 3D camera; and

determining, by the computer, a relative position between the first, thermal camera and the second, 3D camera from the thermal marker locations and the 3D marker locations.

3. The method of claim 2, wherein the identifying the edge of each of the plurality of marker surfaces and the identifying the locations of each of the plurality of marker surfaces comprises:

finding two or more edge pixels lying on a boundary of a respective marker surface;

calculating a direction of a gradient for each found edge pixel; and

calculating the location of the marker surface from the directions of the gradients.

4. The method of claim 2, wherein the identifying the locations of each of the plurality of marker surfaces comprises:

obtaining a two-dimensional (2D) view of the 3D point cloud by rotating the 3D point cloud such that the planar surface of the hybrid phantom is perpendicular to a depth direction of a reference system of the second, 3D camera;

obtaining auxiliary marker surface locations by identifying the locations of the plurality of marker surfaces in the 2D view of the 3D point cloud; and

obtaining the locations of each of the plurality of marker surfaces by performing an inverse rotation of the auxiliary marker surface locations.

5. The method of claim 4, wherein a plane is fitted into the points of the 3D point cloud which represent the planar surface of the hybrid phantom, a transformation describing rotations about axes perpendicular to the depth direction of the reference system of the second camera and making the plane perpendicular to the depth direction of the reference system of the second camera is calculated, the transformation is applied to the 3D point cloud and the inverse of the transformation is applied to the auxiliary marker surface locations.

6. The method of claim 5, wherein points of the 3D point cloud which do not lie on the plane are removed from the 3D point cloud.

7. The method of claim 4, wherein the 2D view of the 3D point cloud is a depth map of the rotated 3D point cloud.

8. The method of claim 2, further comprising:

adjusting the temperature of at least a part of the background surface of the hybrid phantom such that it differs from the ambient temperature.

9. The method of claim 8, wherein the hybrid phantom is cooled using a cooling element or heated using a heating element.

10. The method of claim 8, wherein the hybrid phantom comprises metallic inlays forming the plurality of marker surfaces and the metallic inlays are heated such that their heat dissipates into the background surface surrounding the marker surfaces.

11. A system comprising;

a hybrid phantom comprising;

a planar surface comprising;

a background surface having one of a first property and a second property; and

a plurality of marker surfaces, each marker surface of the plurality of marker surfaces having the other one of the first property and the second property;

wherein the first property involves having a high thermal emissivity of 0.8 or higher and the second property involves having a low thermal emissivity of 0.2 or less, and

wherein one of the first property and the second property further involves being diffuse reflective and the other one of the first property and the second property further involves being specular reflective;

a first, thermal camera adapted to capture a thermal image of the hybrid phantom;

a second, three-dimensional (3D) camera adapted to capture a 3D point cloud of the hybrid phantom; and

a computer configured to;

identify an edge of each of the plurality of marker surfaces in the thermal image captured by the first thermal camera;

generate thermal marker locations by identifying locations of each of the plurality of marker surfaces from the identified edge of each of the plurality of marker surfaces in the thermal image;

generate 3D marker locations by identifying locations of the plurality of marker surfaces from the 3D point cloud captured by the second, 3D camera; and

determine a relative position between the first thermal camera and the second 3D, camera from the generated thermal marker locations and the generated 3D marker locations.

12. The system of claim 11, further comprising a heating element or a cooling element, wherein the heating element is configured to heat the hybrid phantom or the cooling element is configured to cool the hybrid phantom.

13. The system of claim 11, wherein the hybrid phantom further comprises metallic inlays forming the plurality of marker surfaces, and a heating element wherein the heating element is configured to heat the metallic inlays.

14. The system of claim 13, wherein the metallic inlays are adapted such that when heated their heat dissipates into the background surface surrounding the marker surfaces.

15. The system of claim 11, wherein the identifying the edge of each of the plurality marker surfaces and identifying the locations of each of the plurality of marker surfaces comprises:

finding two or more edge pixels lying on a boundary of a respective marker surface;

calculating a direction of a gradient for each found edge pixel; and

calculating the location of the marker surface from the directions of the gradients.

16. The system of claim 11, wherein the identifying the locations of each of the plurality of marker surfaces comprises:

obtaining a two-dimensional (2D) view of the 3D point cloud by rotating the 3D point cloud such that the planar surface of the hybrid phantom is perpendicular to a depth direction of a reference system of the second, 3D camera;

obtaining auxiliary marker surface locations by identifying the locations of the plurality of marker surfaces in the 2D view of the 3D point cloud; and

obtaining the locations of each of the plurality of marker surfaces by performing an inverse rotation of the auxiliary marker surface locations.

17. The system of claim 16, wherein a plane is fitted into the points of the 3D point cloud which represent the planar surface of the hybrid phantom, a transformation describing rotations about axes perpendicular to the depth direction of the reference system of the second camera and making the plane perpendicular to the depth direction of the reference system of the second camera is calculated, the transformation is applied to the 3D point cloud and the inverse of the transformation is applied to the auxiliary marker surface locations.

18. The system of claim 17, wherein points of the 3D point cloud which do not lie on the plane are removed from the 3D point cloud.

19. The system of claim 16, wherein the 2D view of the 3D point cloud is a depth map of the rotated 3D point cloud.