Vector processing architectures for infrared camera electronics

• US 10,070,074 B2
• Filed: 02/05/2016
• Issued: 09/04/2018
• Est. Priority Date: 08/06/2013
• Status: Active Grant

• Current Assignee: Teledyne FLIR LLC (Teledyne Technologies Incorporated)
• Original Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Boulanger, Pierre, Roberts, Randy
• Primary Examiner(s): Kim, Kiho
• Application Number:
  US15/017,448
• Publication Number:
  US 20160156855A1
• Time in Patent Office: 942 Days
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 5/76 Television signal recording

Abstract

Systems and methods are disclosed herein to provide infrared imaging systems with improved electronics architectures. In one embodiment, an infrared imaging system is provided that includes an infrared imaging sensor for capturing infrared image data and a main electronics block for efficiently processing the captured infrared image data. The main electronics block may include a plurality of vector processors each configured to operate on multiple pixels of the infrared image data in parallel to efficiently exploit pixel-level parallelism. Each vector processor may be communicatively coupled to a local memory that provides high bandwidth, low latency access to a portion of the infrared image data for the vector processor to operate on. The main electronics block may also include a general-purpose processor configured to manage data flow to/from the local memories and other system functionalities. The main electronics block may be implemented as a system-on-a-chip.

Claims

1. An infrared imaging system comprising:

an infrared imaging sensor configured to provide infrared image data comprising a plurality of pixels; and

a main electronics block implemented as a system-on-a-chip (SOC) and comprising;

a sensor interface circuit configured to receive the infrared imaging data from the infrared imaging sensor;

a plurality of vector processors configured to operate in parallel on respective vector arrays, wherein each vector processor of the plurality of vector processors comprises vector functional units configured to selectably provide a number of lanes for processing the respective vector arrays associated with the vector processor; and

at least one local memory communicatively coupled to the plurality of vector processors, the at least one local memory being addressable and directly accessible by the plurality of vector processors to store and access at least a portion of the infrared image data comprising the plurality of pixels;

wherein a first vector processor of the plurality of vector processors is configured to execute a first set of vector instructions on the plurality of pixels and accessed from the at least one local memory to perform one or more operations of a video processing chain for the infrared image data; and

wherein a second vector processor of the plurality of vector processors is configured to execute a second set of vector instructions on the plurality of pixels accessed from the at least one local memory to perform one or more operations of video analytics for the infrared image data, the second set of vector instructions being different from the first set of vector instructions.

2. The infrared imaging system of claim 1, wherein:

the at least one local memory comprises a plurality of local memories each assigned to one or more of the vector processors; and

the main electronics block further comprises a general-purpose processor configured to;

selectively enable or disable each of the plurality of vector processors; and

manage data flow to the plurality of local memories.

3. The infrared imaging system of claim 2, wherein the main electronics block further comprises a peripheral interface block configured to facilitate communications between the general-purpose processor and one or more peripheral devices.

4. The infrared imaging system of claim 2, further comprising a global memory communicatively coupled to the main electronics block and configured to store the infrared image data, wherein:

the main electronics block further comprises a direct memory access (DMA) engine responsive to DMA transfer requests from the general-purpose processor; and

the general-purpose processor is configured to manage data flow to the plurality of local memories by requesting the DMA engine to perform a DMA transfer of at least a portion of the infrared image data from the global memory to at least one of the plurality of local memories.

5. The infrared imaging system of claim 2, wherein:

a first local memory of the plurality of local memories is communicatively coupled to the first vector processor;

a second local memory of the plurality of local memories is communicatively coupled to the second vector processor;

the first vector processor is configured to access the plurality of pixels from the first local memory;

the second vector processor is configured to access the plurality of pixels from the second local memory; and

the general-purpose processor is configured to manage data flow to and/or between the first and second local memories.

6. The infrared imaging system of claim 1, wherein:

the first set of vector instructions associated with the first vector processor for performing one or more operations of the video processing chain includes instructions for performing offset correction, gain correction, bad pixel replacement, automatic gain control, and/or optical distortion correction on a group of pixels among the plurality of pixels in parallel.

7. The infrared imaging system of claim 1, wherein

a third vector processor of the plurality of vector processors is configured to execute a third set of vector instructions to perform one or more operations of an image/video resolution enhancement.

8. The infrared imaging system of claim 1, further comprising a visible light imaging sensor configured to provide visible light image data, wherein a third vector processor of the plurality of vector processors is configured to execute a third set of vector instructions to perform one or more operations of infrared and visible light image data blending.

9. The infrared imaging system of claim 1, wherein the at least one local memory is a shared local memory for the plurality of vector processors.

10. The infrared imaging system of claim 1, wherein:

the plurality of pixels is provided as a plurality of groups of pixels;

for each group of pixels, the first vector processor is configured to execute the first set of vector instructions on the group of pixels in parallel with the second vector processor executing the second set of vector instructions on another group of pixels; and

each of the plurality of pixels is processed by both the first vector processor and the second vector processor.

11. A method of providing infrared images, the method comprising:

converting received infrared energy into infrared image data comprising a plurality of pixels;

receiving the infrared image data at a system-on-a-chip (SOC) via a sensor interface circuit of the SOC, wherein the SOC comprises a plurality of vector processors configured to operate in parallel on respective vector arrays and at least one local memory communicatively coupled to the plurality of vector processors, wherein each vector processor of the plurality of vector processors comprises vector functional units configured to selectably provide a number of lanes for processing the respective vector arrays associated with the vector processor;

providing, to at least one vector processor of the plurality of vector processors, the plurality of pixels of the infrared image data by directly addressing and accessing at least a portion of the infrared image data comprising the plurality of pixels from the at least one local memory;

executing a first set of vector instructions by a first vector processor of the plurality of vector processors on the plurality of pixels and accessed from the at least one local memory to perform one or more operations of a video processing for the infrared image data; and

executing a second set of vector instructions by a second vector processor of the plurality of vector processors on the plurality of pixels accessed from the at least one local memory to perform one or more operations of video analytics for the infrared image data, the second set of vector instructions being different from the first set of vector instructions.

12. The method of claim 11, wherein:

the at least one local memory comprises a plurality of local memories each assigned to one or more of the vector processors; and

the method further comprises managing data flow to the plurality of local memories by a general-purpose processor.

13. The method of claim 12, wherein the providing the plurality of pixels comprises:

transferring the plurality of pixels to a first local memory of the plurality of local memories, the first local memory communicatively coupled to the first vector processor; and

transferring the plurality of pixels to a second local memory of the plurality of local memories, the second local memory communicatively coupled to the second vector processor.

14. The method of claim 12, wherein the managing of the data flow to the plurality of local memories comprises transferring, by a direct memory access (DMA) engine, the at least a portion of the infrared image data from a global memory to at least one of the plurality of local memories in response to a DMA transfer request from the general-purpose processor.

15. The method of claim 11, wherein:

the first set of vector instructions executed by the first vector processor to perform one or more operations of the video processing chain includes instructions for performing offset correction, gain correction, bad pixel replacement, automatic gain control, and/or optical distortion correction on a group of pixels among the plurality of pixels in parallel.

16. The method of claim 11, further comprising executing a third set of vector instruction by a third vector processor of the plurality of vector processors to perform one or more operations of an image/video resolution enhancement.

17. The method of claim 11, further comprising:

converting received visible light into visible light image data; and

executing a third set of vector instruction by a third vector processor of the plurality of vector processors to perform one or more operations of infrared and visible light image data blending.