Super resolution and color motion artifact correction in a pulsed color imaging system

• US 9,641,815 B2
• Filed: 03/14/2014
• Issued: 05/02/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Richardson, John, Blanquart, Laurent
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Messmore, Jonathan
• Application Number:
  US14/214,311
• Publication Number:
  US 20140267655A1
• Time in Patent Office: 1,145 Days
• Field of Search: 348 68
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments and associated structures, methods and features. The features of the systems and methods described herein may include providing improved resolution and color reproduction.

Claims

1. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

actuating an emitter to emit a plurality of pulses of electromagnetic radiation, wherein the plurality of pulses comprises a sequence of predetermined wavelengths of electromagnetic radiation to cause illumination within the light deficient environment;

pulsing said emitter at a predetermined interval;

sensing reflected electromagnetic radiation from each of said pulses with a pixel array;

wherein said pixel array is actuated at a sensing interval that corresponds to the pulse interval of said emitter;

creating a sample frame for each pulse of electromagnetic radiation in the sequence, wherein the sample frame comprises one of chrominance or luminance;

providing a plurality of sample frames to create an output frame, wherein the plurality of sample frames comprise a plurality of luminance sample frames and at least one chrominance sample frame;

detecting motion of objects being imaged;

increasing resolution for the pixel array by compensating for the detected motion based at least in part on the luminance sample frames;

wherein increasing resolution comprises;

sensing luminescence of a plurality of neighboring pixels to gather luminance data;

bilinear interpolating the luminance data into a first upscaled data set;

bicubic interpolating the luminance data into a second upscaled data set;

creating a baseline with no interpolation of the luminance data into a third upscaled data set; and

creating a stream of images by combining each output frame, such that there is a constant output frame rate.

2. The method of claim 1, wherein said sensing process comprises sensing luminance and linear sums of luminance plus chrominance in adjacent images in a stream of images forming a video stream.

3. The method of claim 2, further comprising indexing frames within the video stream with a rotating frame index.

4. The method of claim 3, wherein the rotating frame index comprises four counts.

5. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame.

6. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame and following indexed frame.

7. The method of claim 4, further comprising reconstructing a frame for luminance and two frames for chrominance.

8. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a plurality of prior indexed frames and a plurality of latter indexed frames for increased resolution and accuracy.

9. The method of claim 1, wherein the first upscaled data set is used for block matching.

10. The method of claim 1, wherein the second upscaled data set is used for fall back pixel data.

11. The method of claim 1, wherein the third upscaled data set forms the baseline for a resolution-enhanced data set.

12. The method of claim 1, further comprising segmenting data created by the pixel array into segments of pixels and nearest neighbors.

13. The method of claim 12, further comprising shifting each segment of pixels in the x direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the x direction.

14. The method of claim 13, further comprising shifting each segment of pixels in the x direction in sub-pixel increments and comparing to a first up-scaled data set for greater precision of motion detection in the x direction.

15. The method of claim 13, further comprising shifting each segment of pixels in the y direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the y direction.

16. The method of claim 15, further comprising shifting each segment of pixels in the y direction in sub-pixel increments and comparing to a first up-scaled data set for greater precision of motion detection in the y direction.

17. The method of claim 15, further comprising determining the vector of the motion of the object by combining the x and y motion of the object being imaged.

18. The method of claim 17, further comprising estimating motion to combine data from multiple luminance frames into a single, higher resolution luminance frame.

19. The method of claim 18, wherein said process is repeated for every frame containing luminance data in a continuous sequence.

20. The method of claim 19, further comprising utilizing the motion estimation between neighboring luminance frames to infer the motion at the intermediate frames, containing the chrominance data, with respect to luminance frames, by interpolation.

21. The method of claim 20, wherein said process is repeated for every frame containing chrominance data in a continuous sequence.

22. The method of claim 1, further comprising:

actuating the emitter to emit a plurality of pulses of electromagnetic radiation in sequence to cause illumination;

wherein a first pulse is within a first range that is only a portion of an electromagnetic spectrum,

wherein a second pulse is within a second range that is only a portion of an electromagnetic spectrum,

wherein a third pulse is within a third range that is only a portion of an electromagnetic spectrum,

pulsing said pulses at a predetermined interval,

wherein said pixel array is actuated at a first sensing interval that corresponds to the pulse interval of said first pulse,

wherein said pixel array is actuated at a second sensing interval that corresponds to the pulse interval of said second pulse, and

wherein said pixel array is actuated at a third sensing interval that corresponds to the pulse interval of said third pulse.

23. The method of claim 22, wherein a blanking interval is concurrent with a portion of any interval of said first pulse, second pulse, or third pulse.

24. The method of claim 1, wherein the method further comprises correcting for color artifacts due to the pulsing of the emitter and the detected motion, wherein the plurality of sample frames comprise adjacent luminance sample frames and an intermediate chrominance sample frame, and wherein the correcting for color artifacts utilizes a relative motion estimate for the adjacent luminance sample frames to predict motion that occurred for the intermediate chrominance frame relative to the luminance frames to which they become associated during a color fusion to create the output frame.

25. The method of claim 1, wherein said emitter is a laser diode.

26. The method of claim 1, wherein said emitter is a light emitting diode (LED).

27. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

actuating an emitter to emit a plurality of pulses of electromagnetic radiation, wherein the plurality of pulses comprises a sequence of predetermined wavelengths of electromagnetic radiation to cause illumination within the light deficient environment;

pulsing said emitter at a predetermined interval;

sensing reflected electromagnetic radiation from each of said pulses with a pixel array;

wherein said pixel array is actuated at a sensing interval that corresponds to the pulse interval of said emitter;

creating a sample frame for each pulse of electromagnetic radiation in the sequence, wherein the sample frame comprises one of chrominance or luminance;

providing a plurality of sample frames to create an output frame, wherein the plurality of sample frames comprise a plurality of luminance sample frames and at least one chrominance sample frame;

detecting motion of objects being imaged;

increasing resolution for the pixel array by compensating for the detected motion based at least in part on the luminance sample frames;

wherein increasing resolution comprises;

sensing luminescence of a plurality of neighboring pixels to gather luminance data;

bilinear interpolating the luminance data into a first upscaled data set;

bicubic interpolating the luminance data into a second upscaled data set;

creating a baseline with no interpolation of the luminance data into a third upscaled data set;

correcting for color artifacts due to the pulsing of the emitter and the detected motion; and

creating a stream of images by combining each output frame, such that there is a constant output frame rate.

28. The method of claim 27, wherein the first upscaled data set is used for block matching.

29. The method of claim 27, wherein the second upscaled data set is used for fall back pixel data.

30. The method of claim 27, wherein the third upscaled data set forms the basis of a resolution-enhanced data set.

31. The method of claim 27, further comprising segmenting data created by the pixel array into segments of pixels and nearest neighbors.

32. The method of claim 31, further comprising shifting each segment of pixels in the x direction in order to determine motion of an object being imaged in the x direction.

33. The method of claim 32, further comprising shifting each segment of pixels in the y direction in order to determine motion of an object being imaged in the y direction.

34. The method of claim 33, further comprising determining the vector of the motion of the object by combining the x and y motion of the object being imaged.

35. The method of claim 34, further comprising plotting the motion of the object being imaged by accounting for the modulus of pixel differences between frames of the stream of images.

36. The method of claim 27, further comprising:

actuating the emitter to emit a plurality of pulses of electromagnetic radiation in sequence to cause illumination,

wherein a first pulse is within a first range that is only a portion of an electromagnetic spectrum,

wherein a second pulse is within a second range that is only a portion of an electromagnetic spectrum,

wherein a third pulse is within a third range that is only a portion of an electromagnetic spectrum,

pulsing said pulses at a predetermined interval,

wherein said pixel array is actuated at a first sensing interval that corresponds to the pulse interval of said first pulse,

wherein said pixel array is actuated at a second sensing interval that corresponds to the pulse interval of said second pulse, and

wherein said pixel array is actuated at a third sensing interval that corresponds to the pulse interval of said third pulse.

37. The method of claim 36, further comprising:

actuating said emitter to not emit light for a calibration interval, and actuating said pixel array during said calibration interval.

38. The method of claim 37, further comprising:

stopping further pulses if said pixel array senses light during said calibration interval.

39. The method of claim 36, wherein a blanking interval is not concurrent with any interval of said first beam of light, second beam of light, or third beam of light.

40. The method of claim 36, wherein a blanking interval is concurrent with a portion of any interval of said first pulse, second pulse, or third pulse.

41. The method of claim 36, wherein said pulse is of a green visible light spectrum range, and wherein said second pulse is of a red visible light spectrum, and wherein said third pulse is of a blue visible light spectrum.

42. The method of claim 36, wherein one of said pulses is from a non-visible range of electromagnetic spectrum.

43. The method of claim 36, wherein said sensor is configured to sense any of said first pulse, second pulse, and third pulse equally.

44. The method of claim 36, wherein said sensor is configured to sense any spectrum range of said electromagnetic spectrum.

45. The method of claim 27, wherein said emitter is a laser diode.

46. The method of claim 27, wherein said emitter is a light emitting diode (LED).