Apparatus and method for multispectral imaging with three dimensional overlaying

• US 9,654,704 B2
• Filed: 03/13/2014
• Issued: 05/16/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Infrared Integrated Systems, Ltd. (Fortive Corp.)
• Original Assignee: Infrared Integrated Systems, Ltd. (Fortive Corp.)
• Inventors: Holliday, Stuart
• Primary Examiner(s): GUNBERG, EDWIN C
• Application Number:
  US14/208,563
• Publication Number:
  US 20140346359A1
• Time in Patent Office: 1,160 Days
• CPC Class Codes: G01J 1/42 using electric radiation detectors optical or mechanical part G01J1/04; by comparison with a reference light or electric value G01J1/10
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/10 using electric radiation detectors
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/265 Mixing

Abstract

An apparatus and a method for multispectral imaging comprising, representation generator arranged to generate a three dimensional representation of a scene, at least one infrared imaging sensor arranged to obtain an infrared image of the scene, and an image overlaying processor arranged to overlay the infrared image onto the three dimensional representation of the scene to produce an infrared three dimensional representation of the scene.

Claims

1. Apparatus for imaging, comprising:

a representation generator arranged to generate a three dimensional representation of a scene;

an image segmentation process arranged to identify objects located at different depths in the scene;

at least one visible light imaging sensor arranged to obtain a visible light image of the scene;

at least one infrared imaging sensor arranged to obtain an infrared image of the scene; and

an image overlaying processor arranged to;

overlay the infrared image and the visible light image onto the three dimensional representation of the scene to produce an multispectral three dimensional representation of the scene; and

manipulate the overlaid infrared image in the multispectral three dimensional representation of the scene according to the depths of the identified objects such that;

the infrared image is not overlaid onto objects in the scene that do not satisfy a predetermined depth requirement so that a palette of false colors used to represent temperature data in the infrared image is adjusted to extend over the range of temperatures observable at objects satisfying the predetermined depth requirement; and

the visible light image is overlaid onto objects in the scene that do not satisfy the predetermined depth requirement.

2. The apparatus of claim 1, wherein the visible light image and the infrared image are combined to provide a combined multispectral image before being overlaid onto the three dimensional representation of the scene.

3. The apparatus of claim 1, wherein the infrared image in the multispectral three dimensional representation of the scene is manipulated by removing objects located at a depth in the scene greater than a first predetermined depth from the representation of the scene.

4. The apparatus of claim 3, the apparatus further comprising means to identify an object of interest in the scene, wherein the first predetermined depth is based on the depth in the scene at which an identified object of interest is located.

5. The apparatus of claim 1, wherein the infrared image in the multispectral three dimensional representation of the scene is manipulated by removing objects located at a depth in the scene smaller than a second predetermined depth from the representation of the scene.

6. The apparatus of claim 1, wherein the infrared image in the multispectral three dimensional representation of the scene is manipulated by selectively overlaying the infrared image onto objects in dependence on the depths at which the objects are located in the three dimensional representation of the scene.

7. The apparatus of claim 1, the apparatus further comprising means to identify an object of interest in the scene, and wherein the predetermined depth requirement is satisfied by depths that are within a range of depths extending to either side of the location of an identified object of interest.

8. The apparatus of claim 7, wherein the range of depths are centred on the location of an identified object of interest.

9. The apparatus of claim 1, wherein the at least one visible light imaging sensor comprises at least two spaced apart imaging sensors arranged to form a cyclopean camera arrangement;

and the apparatus further comprises an image combiner arranged to combine images from the at least two spaced apart imaging sensors to provide a first cyclopean image of a scene from a virtual first position.

10. The apparatus of claim 9, wherein the at least one infrared imaging sensor provides a second image of the scene from a second position;

and

the first virtual position and the second position of the at least one infrared imaging sensor are arranged to be substantially the same.

11. The apparatus of claim 9, wherein the representation generator comprises a processor which compares the images from the at least two spaced apart imaging sensors and uses stereoscopic techniques to determine the depths and/or distances.

12. The apparatus of claim 1, wherein the representation generator comprises a structured light scanner.

13. The apparatus of claim 12, wherein the at least one visible light imaging sensor views structured light projected by the structured light scanner and provides information regarding the viewed structured light to the representation generator.

14. The apparatus of claim 1, wherein overlaying the infrared image onto the three dimensional representation of the scene comprises raycasting the visible image onto the three dimensional representation of the scene.

15. An imaging method, comprising:

generating a three dimensional representation of a scene using a representation generator;

identifying objects located at different depths in the scene using an image segmentation process;

obtaining an infrared image of the scene using at least one infrared imaging sensor;

obtaining a visible light image of the scene using at least one visible light imaging sensor;

overlaying the infrared image and the visible light image onto the three dimensional representation of the scene to produce an multispectral three dimensional representation of the scene using an image overlaying processor; and

manipulating the infrared image in the multispectral three dimensional representation of the scene according to the depths of the identified objects such that;

the infrared image is not overlaid onto objects in the scene that do not satisfy a predetermined depth requirement so that a palette of false colors used to represent temperature data in the infrared image is adjusted to extend over the range of temperatures observable at objects satisfying the predetermined depth requirement; and

the visible light image is overlaid onto the objects in the scene that do not satisfy the predetermined depth requirement.

16. The method of claim 15, wherein the visible light image and the infrared image are combined to provide a combined multispectral image before being overlaid onto the three dimensional representation of the scene.

17. The method of claim 15, wherein manipulating the infrared image in the multispectral three dimensional representation of the scene comprises removing objects located at a depth in the scene greater than a first predetermined depth from the representation of the scene.

18. The method of claim 17, the apparatus further comprising means to identify an object of interest in the scene, wherein the first predetermined depth is based on the depth in the scene at which an identified object of interest is located.

19. The method of claim 15, wherein manipulating the infrared image in the multispectral three dimensional representation of the scene comprises removing objects located at a depth in the scene smaller than a second predetermined depth from the representation of the scene.

20. The method of claim 15, wherein manipulating the infrared image in the multispectral three dimensional representation of the scene comprises selectively overlaying the infrared image onto objects in dependence on the depths at which the objects are located in the three dimensional representation of the scene.

21. The method of claim 15, the apparatus further comprising means to identify an object of interest in the scene, wherein the predetermined depths are a range of depths extending to either side of the location of an identified object of interest.

22. The method of claim 21, wherein the range of depths are centred on the location of an identified object of interest.

23. The method of claim 15, wherein the at least one visible light imaging sensor comprises at least two spaced apart imaging sensors arranged to form a cyclopean camera arrangement;

and

the method further comprises using an image combiner to combine images from the at least two spaced apart imaging sensors to provide a first cyclopean image of a scene from a first virtual position.

24. The method of claim 23 wherein the at least one infrared imaging sensor provides a second image of the scene from a second position;

and

the first virtual position and the second position of the at least one infrared imaging sensor are arranged to be substantially the same.

25. The method of claim 23, wherein the representation generator comprises a processor which compares the images from the at least two spaced apart imaging sensors and uses stereoscopic techniques to determine the depths and/or distances.

26. The method according claim 15, wherein the representation generator comprises a structured light scanner.

27. The method of claim 26, wherein the at least one visible light imaging sensor views structured light projected by the structured light scanner and provides information regarding the viewed structured light to the representation generator.

28. The method of claim 15, wherein overlaying the infrared image onto the three dimensional representation of the scene comprises raycasting the visible image onto the three dimensional representation of the scene.