SUPER RESOLUTION AND COLOR MOTION ARTIFACT CORRECTION IN A PULSED COLOR IMAGING SYSTEM

• US 20140267655A1
• Filed: 03/14/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Olive Medical Corporation (Johnson & Johnson)
• Inventors: Richardson, John, Blanquart, Laurent
• Granted Patent: US 9,641,815 B2
• Time in Patent Office: Days
• US Class Current: 348/68
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments and associated structures, methods and features. The features of the systems and methods described herein may include providing improved resolution and color reproduction.

Claims

1. A digital imaging method for use with an endo scope in ambient light deficient environments comprising:

actuating an emitter to emit a pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment;

wherein said pulse is within a first wavelength range that comprises a first portion of electromagnetic spectrum;

pulsing said emitter at a predetermined interval;

sensing reflected electromagnetic radiation from said pulse with a pixel array;

wherein said pixel array is actuated at a sensing interval that corresponds to the pulse interval of said emitter;

detecting motion of objects being imaged;

increasing resolution for the pixel array by compensating for the detected motion; and

creating a stream of images by combining a plurality of sensed reflected electromagnetic energies into a frame.

2. The method of claim 1, wherein increasing resolution comprises:

sensing luminescence of a plurality of neighboring pixels to gather luminance data;

bilinear interpolating the luminance data into a first upscaled data set;

bicubic interpolating the luminance data into a second upscaled data set;

creating a baseline with no interpolation of the luminance data into a third upscaled data set.

3. The method of claim 1, wherein said sensing process comprises sensing luminance and linear sums of luminance plus chrominance in adjacent images in a stream of images forming a video stream.

4. The method of claim 3, further comprising indexing frames within the video stream with a rotating frame index.

5. The method of claim 4, wherein the rotating frame index comprises four counts.

6. The method of claim 5, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame.

7. The method of claim 5, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame and following indexed frame.

8. The method of claim 5, further comprising reconstructing a frame for luminance and two frames for chrominance.

9. The method of claim 5, further comprising reconstructing the video stream by combining luminance and chrominance data from a plurality of prior indexed frames and a plurality of latter indexed frames for increased resolution and accuracy.

10. The method of claim 2, wherein the first upscaled data set is used for block matching.

11. The method of claim 2, wherein the second upscaled data set is used for fall back pixel data.

12. The method of claim 2, wherein the third upscaled data set forms the baseline for the resolution-enhanced data set.

13. The method of claim 1, further comprising segmenting data created by the pixel array into segments of pixels and nearest neighbors.

14. The method of claim 13, further comprising shifting each segment of pixels in the x direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the x direction.

15. The method of claim 14, further comprising shifting each segment of pixels in the x direction in sub-pixel increments and comparing to the first up-scaled data set for greater precision of motion detection in the x direction.

16. The method of claim 14, further comprising shifting each segment of pixels in the y direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the y direction.

17. The method of claim 16, further comprising shifting each segment of pixels in the y direction in sub-pixel increments and comparing to the first up-scaled data set for greater precision of motion detection in the y direction.

18. The method of claim 16, further comprising determining the vector of the motion of the object by combining the x and y motion of the of the object being imaged.

19. The method of claim 18, further comprising estimating motion to combine data from multiple luminance frames into a single, higher resolution luminance frame.

20. The method of claim 19, wherein said process is repeated for every frame containing luminance data in a continuous sequence.

21. The method of claim 20, further comprising utilizing the motion estimation between neighboring luminance frames to infer the motion at the intermediate frames, containing the chrominance data, with respect to luminance frames, by interpolation.

22. The method of claim 21, wherein said process is repeated for every frame containing chrominance data in a continuous sequence.

23. The method of claim 1, further comprising:

actuating the emitter to emit a plurality of pulses of electromagnetic radiation in sequence to cause illumination;

wherein a first pulse is within a first range that is only a portion of a electromagnetic spectrum,

wherein a second pulse is within a second range that is only a portion of a electromagnetic spectrum,

wherein a third pulse is within a third range that is only a portion of a electromagnetic spectrum,

pulsing said pulses at a predetermined interval,

wherein said pixel array is actuated at a first sensing interval that corresponds to the pulse interval of said first pulse,

wherein said pixel array is actuated at a second sensing interval that corresponds to the pulse interval of said second pulse, and

wherein said pixel array is actuated at a third sensing interval that corresponds to the pulse interval of said third pulse.

24. The method of claim 23, wherein a blanking interval is concurrent with a portion of any interval of said first pulse, second pulse, third pulse.

25. The method of claim 1, wherein the method further comprises correcting for color artifacts due to the pulsing of the emitter and the detected motion based on the detected motion.

26. The method of claim 1, wherein said emitter is a laser diode.

27. The method of claim 1, wherein said emitter is a light emitting diode (LED).

28. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

actuating an emitter to emit a pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment;

wherein said pulse is within a first wavelength range that comprises a first portion of electromagnetic spectrum;

pulsing said emitter at a predetermined interval;

sensing reflected electromagnetic radiation from said pulse with a pixel array;

wherein said pixel array is actuated at a sensing interval that corresponds to the pulse interval of said emitter;

detecting motion of objects being imaged;

increasing resolution for the pixel array by compensating for the detected motion;

correcting for color artifacts due to the pulsing of the emitter and the detected motion based on the detected motion; and

creating a stream of images by combining a plurality of sensed reflected electromagnetic energies into a frame.

29. The method of claim 28, wherein increasing comprises:

sensing luminescence of a plurality of neighboring pixels to gather luminance data;

bilinear interpolating the luminance data into a first upscaled data set;

bicubic interpolating the luminance data into a second upscaled data set;

creating a baseline with no interpolation of the luminance data into a third upscaled data set.

30. The method of claim 29, wherein the first upscaled data set is used for block matching.

31. The method of claim 29, wherein the second upscaled data set is used for fall back pixel data.

32. The method of claim 29, wherein the third upscaled data set forms the basis of the resolution-enhanced data set.

33. The method of claim 28, further comprising segmenting data created by the pixel array into segments of pixels and nearest neighbors.

34. The method of claim 33, further comprising shifting each segment of pixels in the x direction in order to determine motion of an object being imaged in the x direction.

35. The method of claim 34, further comprising shifting each segment of pixels in the y direction in order to determine motion of an object being imaged in the y direction.

36. The method of claim 35, further comprising determining the vector of the motion of the object by combining the x and y motion of the of the object being imaged.

37. The method of claim 36, further comprising plotting the motion of the object being imaged by accounting for the modulus of pixel differences between frames of the stream of images.

38. The method of claim 28, further comprising:

actuating the emitter to emit a plurality of pulses of electromagnetic radiation in sequence to cause illumination,

wherein a first pulse is within a first range that is only a portion of a electromagnetic spectrum,

wherein a second pulse is within a second range that is only a portion of a electromagnetic spectrum,

wherein a third pulse is within a third range that is only a portion of a electromagnetic spectrum,

pulsing said pulses at a predetermined interval,

wherein said pixel array is actuated at a first sensing interval that corresponds to the pulse interval of said first pulse,

wherein said pixel array is actuated at a second sensing interval that corresponds to the pulse interval of said second pulse, and

wherein said pixel array is actuated at a third sensing interval that corresponds to the pulse interval of said third pulse.

39. The method of claim 38, further comprising:

actuating said emitter to not emit light for a calibration interval, and actuating said pixel array during said calibration interval.

40. The method of claim 38, further comprising:

stopping further pulses if said pixel array senses light during said calibration interval.

41. The method of claim 38, wherein a blanking interval is not concurrent with any interval of said first beam of light, second beam of light, third beam of light.

42. The method of claim 38, wherein a blanking interval is concurrent with a portion of any interval of said first pulse, second pulse, third pulse.

43. The method of claim 38, wherein said pulse is of a green visible light spectrum range, and wherein said second pulse is of a red visible light spectrum, and wherein said third pulse is of a blue visible light spectrum.

44. The method of claim 38, wherein one of said pulses is from a non-visible range of electromagnetic spectrum.

45. The method of claim 38, wherein said sensor is configured to sense any of said first pulse, second pulse, third pulse equally.

46. The method of claim 38, wherein said sensor is configured to sense any spectrum range of said electromagnetic spectrum.

47. The method of claim 28, wherein said emitter is a laser diode.

48. The method of claim 28, wherein said emitter is a light emitting diode (LED).