Multiplexed Fourier ptychography imaging systems and methods

• US 10,652,444 B2
• Filed: 12/04/2015
• Issued: 05/12/2020
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Horstmeyer, Roarke W., Zheng, Guoan, Yang, Changhuei
• Primary Examiner(s): Kir, Albert
• Application Number:
  US14/960,252
• Publication Number:
  US 20160088205A1
• Time in Patent Office: 1,621 Days
• Field of Search: 356445, 359385, 204452, 2502081, 348 80
• CPC Class Codes: G02B 21/002 Scanning microscopes scanning near field optical microscopes G01Q60/18
• CPC Class Codes: G02B 21/365 Control or image processing arrangements for digital or video microscopes G02B21/361, G02B21/362 take precedence
• CPC Class Codes: G02B 21/367 providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
• CPC Class Codes: G02B 27/58 Optics for apodization or superresolution; Optical synthetic aperture systems
• CPC Class Codes: G06V 20/693 Acquisition
• CPC Class Codes: G21K 7/00 Gamma- or X-ray microscopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Certain embodiments pertain to Multiplexed Fourier Ptychographic imaging systems and methods. In one example, the Multiplexed Fourier Ptychographic imaging system includes an LED array configured to illuminate a sequence of LED patterns for illuminating a sample being imaged. The system includes LED circuitry configured to independently control power to turn on multiple LEDs simultaneously in each LED pattern of the array. The system has a light detector that acquires a first set of lower resolution images of the sample each image acquired during exposure time during illumination by a unique LED pattern. The system uses the first set of lower resolution images to generate a second set of lower resolution images associated with each LED in the LED array and iteratively updates overlapping regions in the Fourier domain with the second set of lower resolution images to generate a higher resolution image.

Claims

1. A multiplexed Fourier Ptychographic imaging system comprising:

a light emitting diode (LED) array configured to illuminate a sequence of different LED patterns to provide multiplexed illumination, each LED pattern comprising a plurality of illuminated LEDs, the LED array located to illuminate a sample being imaged;

LED circuitry in electrical communication with the LED array, wherein the LED circuitry is configured to independently control power to turn on multiple LEDs simultaneously in each LED pattern;

a lens configured to collect light issuing from the illuminated sample;

a light detector configured to receive light collected by the lens and to acquire a first set of lower resolution images associated with the sequence of LED patterns, wherein each lower resolution image of the first set is based on light issued from the illuminated sample and received during an exposure time; and

a processor configured to execute instructions to;

a) use the first set of lower resolution images based on multiplexed illumination to generate a second set of lower resolution images, wherein each image of the second set is associated with one LED in the LED array, and

b) generate a higher resolution complex image of the sample using a phase-retrieval operation to recover phase data from the second set of lower resolution images by;

(i) initializing a higher resolution solution in Fourier domain;

(ii) low pass filtering the initial higher resolution solution to generate data in a plurality of regions of the Fourier domain;

(iii) inverse Fourier transforming data in a region of the plurality of regions to generate image data, replacing amplitudes of the image data with the square root of intensity measurements from an image of the second set of lower resolution images associated with an LED corresponding to the region in the Fourier domain to produce updated image data, Fourier transforming the updated image data, and update data in the region of the Fourier domain with the Fourier-transformed updated image data;

(iv) repeating operations in (iii) for each region of the plurality of the regions to determine an updated higher resolution image; and

(v) iteratively determining one or more updated higher resolution solutions by repeating operations in (iv) until a comparison of data derived from a current updated higher resolution solution in the one or more updated higher resolution solutions and data derived from a previous higher resolution solution or the initial higher resolution solution is less than a predefined value,

wherein the higher resolution complex image has higher resolution than the second set of lower resolution images.

2. The multiplexed Fourier Ptychographic imaging system of claim 1, wherein the processor is further configured to send control signals through the LED circuitry to the LED array to turn on multiple LEDs simultaneously in each LED pattern.

3. The multiplexed Fourier Ptychographic imaging system of claim 1, wherein the LED circuitry comprises a wire from each LED in the LED array to control logic of the processor.

4. The multiplexed Fourier Ptychographic imaging system of claim 1, wherein the processor is further configured to execute instructions for determining the sequence of LED patterns and the exposure time.

5. The multiplexed Fourier Ptychographic imaging system of claim 4, wherein the LED patterns and the exposure time are determined based on tuning input.

6. The multiplexed Fourier Ptychographic imaging system of claim 4, wherein the processor is configured to determine the LED patterns and the exposure time based on one or both of a processing time of the Fourier Ptychographic imaging system and a signal-to-noise ratio (SNR) of images acquired during operation.

7. The multiplexed Fourier Ptychographic imaging system of claim 4, wherein the processor is further configured to execute instructions for determining the sequence of LED patterns and the exposure time based on one or both of a maximum processing time and a minimum SNR.

8. The multiplexed Fourier Ptychographic imaging system of claim 1, wherein each LED pattern in the sequence is a Hadamard pattern.

9. The multiplexed Fourier Ptychographic imaging system of claim 8, wherein each LED pattern has a number of LEDs equal to more than half the total number of LEDs in the LED array.

10. An LED assembly in a multiplexed Fourier Ptychographic imaging system, the LED array assembly comprising:

an LED array configured to sequentially illuminate different LED patterns of a sequence of LED patterns to provide multiplexed illumination, the LED array located to illuminate a sample being imaged; and

LED circuitry in electrical communication with the LED array, wherein the LED circuitry is configured to independently control power to turn on multiple LEDs simultaneously in each LED pattern, and

instructions that determine the sequence of LED patterns and the exposure time based on tuning input including a processing time and a signal-to-noise ratio (SNR), wherein the processing time is for using a first set of lower resolution images captured during the multiplexed illumination to generate a second set of lower resolution images, each image of the second set associated with one LED in the LED array, and for generating a higher resolution complex image of the sample using a phase-retrieval operation to recover phase data from the second set of lower resolution images by;

(i) initializing a higher resolution solution in Fourier domain;

(ii) low pass filtering the initial higher resolution solution to generate data in a plurality of regions of the Fourier domain;

(iii) inverse Fourier transforming data in a region of the plurality of regions to generate image data, replacing amplitudes of the image data with the square root of intensity measurements from an image of the second set of lower resolution images associated with an LED corresponding to the region in the Fourier domain to produce updated image data, Fourier transforming the updated image data, and update data in the region of the Fourier domain with the Fourier-transformed updated image data; and

(iv) repeating operations in (iii) for each region of the plurality of the regions to determine an updated higher resolution image; and

(v) iteratively determining one or more updated higher resolution solutions by repeating operations in (iv) until a comparison of data derived from a current updated higher resolution solution in the one or more updated higher resolution solutions and data derived from a previous higher resolution solution or the initial higher resolution solution is less than a predefined value,

wherein the higher resolution complex image has higher resolution than the second set of lower resolution images.

11. The LED array assembly of claim 10, wherein the LED array receives control signals to illuminate the sequence of LED patterns from a controller through the LED circuity.

12. The LED array assembly of claim 11, wherein the LED circuitry comprises a wire from each LED in the LED array to control logic of a processor.

13. The LED array assembly of claim 11, wherein the control signals for illuminating the sequence of LED patterns is based on tuning input of one or both of a maximum processing time and a minimum SNR.

14. A multiplexing Fourier Ptychographic imaging method comprising:

multiplexed illuminating of a sample by a sequence of light emitting diode (LED) patterns using an LED array;

collecting light issuing from the sample using a lens;

acquiring a first set of low resolution images of the sample using a light detector receiving light collected by the lens, each low resolution image of the first set captured during an exposure time while one of the LED patterns is illuminated, wherein each low resolution image of the first set is based on light issuing from the sample;

generating a second set of low resolution images of the sample using the first set, wherein each low resolution image of the second set is associated with a single LED in the LED array patterns; and

reconstructing a high resolution complex image of the sample using a phase-retrieval operation to recover phase data from the second set of low resolution images by;

(i) initializing a higher resolution solution in Fourier domain;

(ii) low pass filtering the initial higher resolution solution to generate data in a plurality of regions;

(iii) inverse Fourier transforming data in a region of the plurality of regions to generate image data, replacing amplitudes of the image data with the square root of intensity measurements from an image of the second set of lower resolution images associated with an LED corresponding to the region in the Fourier domain to produce updated image data, Fourier transforming the updated image data, and replacing data in the region with the Fourier-transformed updated image data;

(iv) repeating operations in (iii) for each region of the plurality of the regions to determine an updated higher resolution image; and

(v) iteratively determining one or more updated higher resolution solutions by by repeating operations in (iv) until a comparison of data derived from a current updated higher resolution solution in the one or more updated higher resolution solutions and data derived from a previous higher resolution solution or the initial higher resolution solution is less than a predefined value,

wherein the higher resolution complex image has higher resolution than the second set of lower resolution images.

15. The multiplexing Fourier Ptychographic imaging method of claim 14, further comprising selecting the LED patterns and the exposure time based on one or both of a processing time and a signal-to-noise ratio.

16. The multiplexing Fourier Ptychographic imaging method of claim 15, further comprising receiving tuning input that includes the processing time or is related to the processing time.

17. The multiplexing Fourier Ptychographic imaging method of claim 14, wherein multiplexed illuminating of the sample with LED patterns comprises receiving control signals at the LED array to independently turn on each LED of each of the sequence of LED patterns.

18. The multiplexing Fourier Ptychographic imaging method of claim 15, wherein selecting the LED patterns and the exposure time comprises selecting instructions for sending control signals to the LED array for illuminating the sequence of LED patterns.