True color infrared photography and video

• US 6,792,136 B1
• Filed: 11/07/2000
• Issued: 09/14/2004
• Est. Priority Date: 11/07/2000
• Status: Expired due to Fees

• Current Assignee: Northrop Grumman Corporation
• Original Assignee: TRW Limited (Zeppelin - Stiftung der Stadt Friedrichshafen)
• Inventors: Niesen, Joseph W.
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): SUKHAPHADHANA, CHRISTOPHER T
• Application Number:
  US09/708,149
• Time in Patent Office: 1,407 Days
• Field of Search: 382/100, 382/103, 382/115, 382/118, 382/162, 382/164, 382/165, 382/173, 382/181, 348/29, 348/32, 348/33, 348/43, 348/152, 348/153, 348/161, 348/164, 348/169, 250/330, 342/179
• US Class Current: 382/118
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

A method beginning at (10) is provided for creating high-fidelity visible coloring from infrared images of a scene under surveillance. The infrared images captured at (12) are analyzed at (14) to determine if an object, such as a face, is identifiable within the image. If an object is identifiable at (16) the object features are compared to a plurality of stored object features at (20). If there is a match at (22), the color characteristics of the object are obtained at (24) and the object is colored at (26) based on the stored database feature information. If there is no match at (22) or identifiable object at (16) and object color cannot be identified at (23), the image is analyzed at (28) to determine if a pattern, such as clothing, is identifiable within the image. If a pattern is identifiable at (30), the color characteristics of the pattern are obtained at (34) and the pattern is colored at (36) according to infrared reflectance characterization in conjunction with the stored pattern information. If no pattern is identifiable at (30), the non-pattern and non-feature containing portions of the image are colored at (38) according to infrared reflectance characterization.

Claims

1. A method of producing a true color image of a scene and subject under surveillance comprising the steps of:

capturing an infrared image of said scene and subject over different spectral bands;

applying a face recognition program to said image to determine if a face of said subject is identifiable in said image;

segmenting said image into a face containing portion and a non-face containing portion if said face is identifiable in said image;

comparing said face containing portion to a plurality of stored facial images;

coloring said face containing portion of said image according to stored facial characteristics if said face containing portion matches a set characteristics in at least one of said plurality of stored facial images; and

determining whether true color is identifiable in said face containing portion of said image if said face containing portion does not match any set of characteristics of said plurality of stored facial images.

2. The method of claim 1 further comprising the step of:

coloring said non-face containing portion of said image according to infrared reflectance characterization.

3. The method of claim 2 wherein said step of coloring said non-face containing portion of said image according to infrared reflectance characterization further comprises applying multispectral imaging to said image.

4. The method of claim 1 further comprising the step of:

applying a pattern recognition program to said image to determine if a pattern is identifiable in said image;

segmenting said image into a pattern containing portion and a non-pattern containing portion if said pattern is identifiable in said image; and

coloring said pattern containing portion of said image according to infrared reflectance characterization in conjunction with stored pattern features.

5. The method of claim 4 further comprising the step of:

coloring said non-pattern containing portion of said image according to infrared reflectance characterization.

6. The method of claim 5 wherein said step of coloring said non-pattern containing portion of said image according to infrared reflectance characterization further comprises applying multispectral imaging to said image.

7. The method of claim 4 wherein said step of applying a pattern recognition program to said image to determine if a pattern is identifiable in said image further comprises looking for one of the group including a face, clothing, painted objects, natural objects, and buildings in said image.

8. The method of claim 1 wherein said step of coloring said face containing portion of said image according to stored facial characteristics further comprises obtaining at least one of the group including skin color, skin tone, hair color, eye color, eye separation and scarring information from a database.

9. The method of claim 1 further comprising assigning true colors to said face containing portion if said true color is identifiable in said face containing portion of said image.

10. A method of producing a true color image of a subject and scene under surveillance comprising the steps of:

capturing an infrared image of the said subject and scene over different spectral bands;

applying a face recognition program to said image to determine if a face is identifiable in said image;

applying a pattern recognition program to said image to determine if a pattern is identifiable in said image;

segmenting said image into a face containing portion if said face is identifiable, a pattern containing portion if said pattern is identifiable, and a non-face/non-pattern containing portion;

comparing said face containing portion of said image to a plurality of facial images stored in a database;

coloring said face containing portion of said image according to stored facial information if said face containing portion matches a set of characteristics in at least one of said plurality of facial images in said database;

determining whether true color is identifiable in said face containing portion of said image if said face containing portion does not match any set of characteristics of said plurality of stored facial images;

coloring said pattern containing portion of said image using multispectral imaging in combination with stored pattern information; and

coloring said non-face/non-pattern containing portion of said image according to infrared reflectance characterization.

11. The method of claim 10 wherein said steps of coloring said non-face/non-pattern containing portion of said image according to infrared reflectance characterization and coloring said pattern containing portion of said image according to infrared reflectance characterization in combination with stored pattern information further comprises applying multispectral imaging to said image.

12. The method of claim 10 further comprising assigning true colors to said face containing portion if said true color is identifiable in said face containing portion of said image.