3D IMAGING METHOD AND SYSTEM

• US 20160006951A1
• Filed: 02/25/2014
• Published: 01/07/2016
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Original Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Inventors: MOGHADAM, Peyman
• Granted Patent: US 9,986,176 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G03B 35/02 by sequential recording
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A system for generating a three-dimensional model of an object, the system including a portable hand-held imaging device including a housing, a plurality of sensors attached to the housing and at least one electronic processing device coupled to the plurality of sensors. The electronic processing device determines from the plurality of sensors, whilst the imaging device is in at least two different poses relative to the object, range data indicative of a range of the object and thermal infrared data indicative of a thermal infrared image of the object, generates a three-dimensional model based at least partially upon the range data from the at least two different poses and associates model data derived from the thermal infrared data from the at least two different poses with the three dimensional model to thereby provide a three dimensional thermal infrared model of the object.

Claims

1. A system for generating a three-dimensional model of an object, the system including:

a portable hand-held imaging device including;

a housing;

a plurality of sensors attached to the housing;

at least one electronic processing device coupled to the plurality of sensors, wherein the electronic processing device;

determines from the plurality of sensors and whilst the imaging device is in at least two different poses relative to the object;

range data indicative of a range of the object;

thermal infrared data indicative of a thermal infrared image of the object;

generates a three-dimensional model based at least partially upon the range data from the at least two different poses; and

associates model data derived from the thermal infrared data from the at least two different poses with the three dimensional model to thereby provide a three dimensional thermal infrared model of the object.

2. The system of claim 1, wherein the electronic processing device:

determines colour data indicative of a visible image of the object; and

associates model data derived from the visible image data from the at least two different poses with the three dimensional model to thereby provide a three dimensional colour model of the object.

3. The system of claim 2, wherein the electronic processing device generates a three dimensional fused thermal infrared and colour model of the object.

4. The system of claim 2, wherein the system includes:

a range sensor attached to the housing that senses a range of the object;

a visible light image sensor attached to the housing that senses a visible image of the object; and

a thermal infrared image sensor attached to the housing that senses a thermal infrared image of the object.

5. The system of claim 1, wherein the electronic processing device;

determines a relative pose of the hand-held device; and

,

uses the relative pose to generate the three dimensional model.

6. The system of claim 5, wherein the electronic processing device determines the relative pose of the hand-held device using at least one of range data, image data and thermal infrared data from at least some of the different poses.

7. The system of claim 6, wherein the electronic processing device:

estimates a timing skew between the range sensor and at least one of a thermal infrared image sensor and a visible light image sensor; and

estimates a pose associated with the image data and/or the thermal infrared data based upon the skew.

8. The system of claim 7, wherein the electronic processing device:

determines a device pose using at least one of the range sensor and a pose sensor;

determines a path of movement of the hand-held device;

determines a visible light image sensor pose and a thermal infrared image sensor pose based on the device pose and the path of movement.

9. The system of claim 5, wherein the system includes a pose sensor and wherein the electronic processing device determines the relative pose using pose data from the pose sensor.

10. The system of claim 9, wherein the pose sensor is at least one of an orientation sensor and an inertial measurement unit.

11. The system of claim 1, wherein the electronic processing device:

selects an object surface part;

identifies images of the selected object surface part whilst the imaging device is in at least two different poses relative to the object;

identifies at least one image pixel corresponding to the selected object surface part for each identified image; and

,

uses the identified image pixels to determine the model data.

12. The system of claim 11, wherein for a thermal infrared image the model data includes model thermal infrared data indicative of a temperature and for a visible image the model data includes model colour data indicative of a colour.

13. The system of claim 11, wherein the electronic processing device associates the data with a voxel in the three dimensional model corresponding to the selected object surface part.

14. The system of claim 11, wherein the selected object surface part is a vertex.

15. The system of claim 11, wherein the electronic processing device determines the model data using a weighted sum of data associated with the identified pixels.

16. The system of claim 15, wherein the electronic processing device determines the weighted sum using a confidence score associated with the identified pixels.

17. The system of claim 16, wherein the electronic processing device determines the confidence score based on factors including at least one of:

a velocity of a sensor when the image was captured;

a position of the one or more pixels in a field of view of the sensor;

an angle between a ray and an object surface, the ray extending from the object surface part to the sensor; and

,

the range when the image was captured.

18. The system of claim 17, wherein for a thermal infrared image, the electronic processing device determines the confidence score based on a temperature value associated with the identified pixels.

19. A method of generating a three-dimensional model of an object, the method including, in an electronic processing device:

determining from the plurality of sensors and whilst an imaging device is in at least two different poses relative to the object;

range data indicative of a range of the object;

thermal infrared data indicative of a thermal infrared image of the object;

generating a three-dimensional model based at least partially upon the range data from the at least two different poses; and

associating model data derived from the thermal infrared data from the at least two different poses with the three dimensional model to thereby provide a three dimensional thermal infrared model of the object.

20. A system for generating a three-dimensional model of an object, the system including:

a portable imaging device including;

a housing;

a range sensor attached to the housing;

a visible light image sensor attached to the housing; and

a thermal infrared image sensor attached to the housing, wherein the range sensor, the visible light image sensor and the thermal infrared image sensor have overlapping fields of view;

at least one processor coupled to the range sensor, the visible light image sensor and the thermal infrared image sensor of the portable imaging device; and

a memory, coupled to the at least one processor, including instruction code executable by the processor for;

receiving range data from the range sensor, image data from the visible light image sensor, and thermal infrared data from the thermal infrared image sensor;

generating a three-dimensional model based upon the range data captured from several positions; and

associating data from the image data and the thermal infrared data, from at least two positions, with the three dimensional model.

21. The system of claim 20, wherein associating the image data and the thermal infrared data with the three-dimensional model comprises estimating a timing skew between the range sensor and at least one of the visible light image sensor and the thermal infrared image sensor, and estimating a pose associated with the image data and/or the thermal infrared data based upon the skew.

22. The system of claim 20, wherein the thermal infrared data is associated with the three-dimensional model based at least partly upon an angle of incidence.

23. The system of claim 20, wherein the portable imaging device further comprises an orientation sensor.

24. The system of claim 20, wherein the system further comprises at least one environmental sensor, wherein the thermal infrared data is associated with the three-dimensional model based at least partly upon data from the at least one environmental sensor.

25. The system of claim 24, wherein the at least one environmental sensor comprises a carbon dioxide sensor, a humidity sensor and/or a temperature sensor.

26. The system of claim 20, wherein the system further comprises an audio sensor for recording an operator'"'"'s voice during operation of the system.

27. The system of claim 20, wherein the housing of the portable imaging device comprises an elongate portion, wherein the range sensor, visible light image sensor and thermal infrared image sensor are attached to an end of the elongate portion.

28. The system of claim 20, wherein the housing of the portable imaging device comprises a trigger for activating at least one of the range sensor, the visible light image sensor and the thermal infrared image sensor.

29. The system of claim 20, wherein the at least one processor is coupled to the range sensor, visible light image sensor and thermal infrared image sensor of the portable imaging device at least in part by a wireless data communication link.

30. The system of claim 20, wherein the thermal infrared image sensor comprises a thermal infrared sensor.

31. The system of claim 20, wherein the memory further includes instruction code executable by the processor for:

generating an output image corresponding to a view of the three-dimensional model, the output image including at least part of the associated image data superimposed on the model and at least part of the associated thermal infrared data superimposed on the model.

32. The system of claim 20, wherein the output image is generated by rendering at least part of the associated visible light image data on a first portion of the model and at least part of the associated thermal infrared data on a second portion of the model.

33. The system of claim 20, wherein the three-dimensional model corresponds to at least a part of a person'"'"'s body and the associated image data includes skin tones of the person.

34. The system of claim 20, wherein the memory further includes instruction code executable by the processor for refining the three-dimension model based upon the image data and/or the thermal infrared data.

35. The system of claim 20, wherein the a portable imaging device further includes a display, for displaying data captured from one or more of the range sensor, visible light image sensor and thermal infrared image sensor, in real time as it is captured.

36. A method of generating a three-dimensional model of an object, the method including:

receiving, from a range sensor, a visible light image sensor and an thermal infrared image sensor, range data, image data and thermal infrared data, wherein the range sensor, the visible light image sensor and the thermal infrared image sensor have overlapping fields of view, and wherein the range data, image data and thermal infrared data each correspond to data from at least two different positions of the range sensor, visible light image sensor and thermal infrared image sensor;

generating a three-dimensional model based upon the range data; and

associating data from the image data and the thermal infrared data, corresponding to the at least two different positions, with the three dimensional model.

37. The method of claim 36, wherein the at least two different positions comprise arbitrary positions, wherein at least a part of an object visible in a first position of the at least two different positions is visible in a second position of the at least two positions.

38. The method of claim 37, wherein the arbitrary positions comprise overlapping views caused by movement of the range sensor, the visible light image sensor and the thermal infrared image sensor across the object.

39. The method of claim 36, further comprising generating an output image corresponding to a view of the three dimensional model, the output image including at least part of the associated image data superimposed on the model and at least part of the associated thermal infrared data superimposed on the model.