Super resolution and color motion artifact correction in a pulsed color imaging system

• US 10,205,877 B2
• Filed: 05/01/2017
• Issued: 02/12/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Richardson, John, Blanquart, Laurent
• Primary Examiner(s): Messmore, Jonathan R
• Application Number:
  US15/583,893
• Publication Number:
  US 20170230574A1
• Time in Patent Office: 652 Days
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments and associated structures, methods and features. The features of the systems and methods described herein may include providing improved resolution and color reproduction.

Claims

1. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

actuating an emitter to emit a pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment;

wherein said pulse is within a first wavelength range that comprises a first portion of electromagnetic spectrum;

pulsing said emitter at a predetermined interval;

sensing reflected electromagnetic radiation from said pulse with a pixel array;

wherein said pixel array is actuated at a sensing interval that corresponds to the pulse interval of said emitter;

detecting motion of objects being imaged;

increasing resolution for the pixel array by compensating for the detected motion by;

sensing luminescence of a plurality of neighboring pixels to gather luminance data,

bilinear interpolating the luminance data into a first upscaled data set,

bicubic interpolating the luminance data into a second upscaled data set, and

creating a baseline with no interpolation of the luminance data into a third upscaled data set; and

creating a stream of images by combining a plurality of sensed reflected electromagnetic energies into a frame.

2. The method of claim 1, wherein said sensing process comprises sensing luminance and linear sums of luminance plus chrominance in adjacent images in a stream of images forming a video stream.

3. The method of claim 2, further comprising indexing frames within the video stream with a rotating frame index.

4. The method of claim 3, wherein the rotating frame index comprises four counts.

5. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame.

6. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a prior indexed frame and following indexed frame.

7. The method of claim 4, further comprising reconstructing a frame for luminance and two frames for chrominance.

8. The method of claim 4, further comprising reconstructing the video stream by combining luminance and chrominance data from a plurality of prior indexed frames and a plurality of latter indexed frames for increased resolution and accuracy.

9. The method of claim 1, wherein the first upscaled data set is used for block matching.

10. The method of claim 1, wherein the second upscaled data set is used for fall back pixel data.

11. The method of claim 1, wherein the third upscaled data set forms the baseline for the resolution-enhanced data set.

12. The method of claim 1, further comprising segmenting data created by the pixel array into segments of pixels and nearest neighbors.

13. The method of claim 12, further comprising shifting each segment of pixels in the x direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the x direction.

14. The method of claim 13, further comprising shifting each segment of pixels in the x direction in sub-pixel increments and comparing to the first up-scaled data set for greater precision of motion detection in the x direction.

15. The method of claim 13, further comprising shifting each segment of pixels in the y direction and comparing with a neighboring frame at the same resolution, in order to determine motion of an object being imaged in the y direction.

16. The method of claim 15, further comprising shifting each segment of pixels in the y direction in sub-pixel increments and comparing to the first up-scaled data set for greater precision of motion detection in the y direction.

17. The method of claim 15, further comprising determining the vector of the motion of the object by combining the x and y motion of the of the object being imaged.

18. The method of claim 17, further comprising estimating motion to combine data from multiple luminance frames into a single, higher resolution luminance frame.

19. The method of claim 18, wherein said process is repeated for every frame containing luminance data in a continuous sequence.