FOVEATED IMAGING SYSTEM AND METHOD

• US 20140247372A1
• Filed: 04/11/2013
• Published: 09/04/2014
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Byren, Robert W.
• Granted Patent: US 9,124,808 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof

Abstract

A time-varying image is synthesized having at least one time-varying region of interest therewithin. The image is spatially modulated with an array of modulating pixels to form a modulated image. The array of modulating pixels has a modulator resolution. The modulated image is sensed with an array of sensor pixels. The array of sensor pixels has a sensor resolution. The modulator resolution is finer than the sensor resolution. Each sensor pixel corresponds to a plurality of modulating pixels. Said sensor pixel and said plurality of modulating pixels subtend the same region in the image. Outside the region of interest, the modulated image is synthesized at the sensor resolution and at a sensor video frame rate. Inside the region of interest, the modulated image is synthesized at the modulator resolution and at an effective synthesized video frame rate that is less than sensor video frame rate.

Claims

1. A method of sensing a time-varying image having at least one time-varying region of interest therewithin, the method comprising:

spatially modulating the time-varying image with an array of modulating pixels to form a modulated image, the array of modulating pixels having a modulator resolution; and

sensing the modulated image with an array of sensor pixels, the array of sensor pixels having a sensor resolution, the modulator resolution being finer than the sensor resolution;

wherein each sensor pixel corresponds to a plurality of modulating pixels, said sensor pixel and said plurality of modulating pixels subtending the same region in the time-varying image;

wherein outside the region of interest, the modulated image is synthesized at the sensor resolution and at a sensor video frame rate;

wherein inside the region of interest, the modulated image is synthesized at the modulator resolution and at an effective synthesized video frame rate that is less than the sensor video frame rate.

2. The method of claim 1,

wherein each modulating pixel is switchable between an on state and an off state;

wherein the on state comprises directing at least a portion of time-varying image light, incident on said modulating pixel, to the modulated image; and

wherein the off state comprises blocking the time-varying image light, incident on said modulating pixel.

3. The method of claim 2, wherein inside the region of interest, said plurality of modulating pixels are modulated in an orthogonal sequence of frames.

4. The method of claim 3, wherein in one frame of the orthogonal sequence, all the modulating pixels in the plurality are in the on state.

5. The method of claim 3, wherein in all frames of the orthogonal sequence, at least two of the modulating pixels in the plurality are in the on state.

6. A method, comprising:

forming an unmodulated image of a time-varying scene;

modulating the unmodulated image with a multi-pixel spatial modulator to form a modulated image of the time-varying scene, the multi-pixel spatial modulator including a plurality of modulator pixels, each modulator pixel being switchable between an on state and an off state;

each modulator pixel in an on state contributing to the modulated image at a location of the modulator pixel with an optical power proportional to an optical power of the unmodulated image incident on the respective modulator pixel;

sensing the modulated image with a multi-pixel sensor, the sensor including a plurality of sensor pixels, the sensor pixels being fewer in number than the modulator pixels, the modulated image being synthesized at a first resolution and a first video frame rate;

identifying a region of interest within the time-varying scene; and

sensing the region of interest with a second resolution finer than the first resolution, and a second video frame rate slower than the first video frame rate.

7. The method of claim 6,

wherein the first resolution corresponds to the resolution of the multi-pixel sensor;

and

wherein the second resolution corresponds to the resolution of multi-pixel spatial modulator.

8. The method of claim 6, wherein each sensor pixel corresponds with a plurality of the modulator pixels, said sensor pixel and said plurality of modulator pixels occupying the same position within the time-varying scene.

9. The method of claim 8, wherein within the region of interest, said plurality of modulator pixels are switched on and off in an orthogonal sequence.

10. The method of claim 9,

wherein the number of modulator pixels in said plurality is X;

wherein within the region of interest, said sensor pixel records X values of optical power, the X values of optical power being orthogonal combinations of optical power from the X modulator pixels; and

wherein the optical powers of the individual modulator pixels are determined from the orthogonal combinations of optical power from the modulator pixels.

11. The method of claim 10, wherein in one of the orthogonal combinations, all the modulator pixels in the plurality are switched on.

12. The method of claim 10,

wherein the optical powers of the individual modulator pixels are determined at second video frame rate;

and

wherein the second video frame rate equals the first video frame rate, divided by X.

13. The method of claim 8,

wherein outside the region of interest, said plurality of modulator pixels are all switched on;

and

wherein outside the region of interest, the optical powers of the individual modulator pixels are combined onto the same sensor pixel and are indistinguishable from one another.

14. The method of claim 6,

wherein the unmodulated image of the time-varying scene is formed with an imaging optic disposed between the time-varying scene and the unmodulated image;

and

wherein the modulated image of the time-varying scene is formed with a reimaging optic disposed between the multi-pixel spatial modulator and the modulated image.

15. A system for imaging a scene, comprising:

an imaging optic configured to collect light from the scene, to form an unmodulated incident beam from the collected light, and to form an unmodulated image of the scene with the unmodulated incident beam;

a spatial light modulator configured to receive the unmodulated incident beam, to receive the unmodulated image thereon, and to form a modulated exiting beam, the spatial light modulator including an array of light modulating elements switchable between a first state and a second state, the spatial light modulator directing portions of the unmodulated incident beam that strike light modulating elements in the first state into the modulated exiting beam, the spatial light modulator blocking portions of the unmodulated incident beam that strike light modulating elements in the second state from entering the modulated exiting beam;

a reimaging optic configured to collect the modulated exiting beam, and to form a spatially modulated image from the modulated exiting beam, the spatially modulated image resembling the scene in areas corresponding to light modulating elements in the first state;

a sensor configured to receive the spatially modulated image thereon, the sensor including an array of sensor elements, the sensor elements being fewer in number than the light modulating elements, the sensor producing an electrical output signal representative of the spatially modulated image;

a processing element configured to receive the electrical output signal and switch the light modulating elements between the first state and the second state;

wherein when at least one region of interest is identified in the scene, the processing element switches the light modulating elements to produce an increase in resolution in the at least one region of interest, and a decrease in video frame rate in the at least one region of interest.

16. The system of claim 15, wherein the at least one region of interest is identified automatically from the spatially modulated image.

17. The system of claim 15, wherein the at least one region of interest is identified by a user from observation of the spatially modulated image.

18. The system of claim 15, wherein the spatial light modulator is a digital micromirror device.

19. The system of claim 15, wherein the imaging optic and the reimaging optic are lenses.

20. The system of claim 15, wherein the sensor elements are independently addressable.

21. The system of claim 15, wherein the processing element comprises a general purpose computer, a tablet computing device, a mobile computing device, an ASIC, a microprocessor, a FPGA, or any combination thereof.