Single image sensor control for capturing mixed mode images

• US 10,201,266 B2
• Filed: 06/03/2017
• Issued: 02/12/2019
• Est. Priority Date: 10/16/2015
• Status: Active Grant

• Current Assignee: CapsoVision, Inc.
• Original Assignee: CapsoVision, Inc.
• Inventors: Wang, Kang-Huai, Hadley, Mark, Lee, Chung-Ta, Lin, Chang-Wei
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US15/613,205
• Publication Number:
  US 20170265730A1
• Time in Patent Office: 619 Days
• Field of Search: 348 46
• CPC Class Codes: A61B 1/000095 for image enhancement
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/041 Capsule endoscopes for imaging
• CPC Class Codes: A61B 1/045 Control thereof
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2201/0079 Medical imaging device
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures

Abstract

A method and apparatus for capturing an image sequence using a capsule camera are disclosed. According to the present invention, a first energy-based frame time for the special images is determined based on first light energy perceived by the image sensor and a second energy-based frame time for the regular images is determined based on second light energy perceived by the image sensor. The capsule camera captures the image sequence comprising one or more sets of mixed-type images by configuring the capsule camera to cause a mixed-frame distance between the first energy-based frame time for one special image in a target set of mixed-type images and the second energy-based frame time for one regular image in the target set of mixed-type images smaller than an average frame period.

Claims

1. A method of capturing an image sequence using a capsule camera when the capsule camera travels through a human gastrointestinal tract after the capsule camera is swallowed by a patient, wherein the capsule camera comprises an image sensor and is capable of capturing at least two mixed-type images corresponding to special images and regular images, the method comprising:

determining one first energy-based frame time for the special images based on first light energy perceived by the image sensor;

determining one second energy-based frame time for the regular images based on second light energy perceived by the image sensor;

capturing the image sequence comprising one or more sets of mixed-type images by configuring the capsule camera to cause a mixed-frame time distance between said one first energy-based frame time for one special image in a target set of mixed-type images and said one second energy-based frame time for one regular image in the target set of mixed-type images smaller than an overall average frame period, wherein each set of mixed-type images comprises at least one special image and one regular image; and

providing or outputting the image sequence captured for further processing or viewing; and

wherein the special images are used to control capturing the regular images or used to derive or provide associated information for the regular images; and

wherein an overall average frame rate is substantially lower than a capable frame rate of the capsule camera, and wherein the overall average frame rate is calculated as a sum of individual frame rates and each individual frame rate is associated with one type of said least two mixed-type images.

2. The method of claim 1, wherein the first light energy perceived by the image sensor is determined by accumulating first illumination power from a light source over a first integration period of the image sensor for the special images and the second light energy perceived by the image sensor is determined by accumulating second illumination power from the light source over a second integration period of the image sensor for the regular images.

3. The method of claim 1, wherein said one first energy-based frame time is determined at a first time instance when the first light energy exceeds a first energy threshold for a target special image and said one second energy-based frame time is determined at a second time instance when the second light energy exceeds a second energy threshold for a target regular image.

4. The method of claim 3, wherein the first energy threshold is set to a half of total accumulated first light energy over a first integration period of the image sensor and the second energy threshold is set to a half of total accumulated second light energy over a second integration period of the image sensor.

5. The method of claim 1, wherein the mixed-frame time distance is smaller than 90 percent of an overall average frame period corresponding to an inverse of the overall average frame rate.

6. The method of claim 1, wherein the special images are captured with a lower quality.

7. The method of claim 6, wherein the lower quality corresponds to lower image dynamic range, lower spatial resolution, smaller image size, or a combination thereof.

8. The method of claim 1, wherein the special images correspond to structured-light images (SLIs) used to derive depth information for associated regular images.

9. The method of claim 8, wherein each set of mixed-type images consists of one structured-light image (SLI) and one regular image.

10. The method of claim 9, wherein said one structured-light image (SLI) is captured prior to said one regular image.

11. The method of claim 9, wherein said one structured-light image (SLI) is captured after said one regular image.

12. The method of claim 8, wherein each set of mixed-type images consists of one structured-light image (SLI) and two regular sub-images, and wherein the structured-light image is captured between the two regular sub-images and the mixed-frame time distance is derived based on two mixed-frame time distances associated with the structured-light image and the two regular sub-images respectively.

13. The method of claim 12, wherein the two regular sub-images are combined to form one regular image.

14. The method of claim 1, wherein the special images correspond to monitor-mode images used to control capturing the regular images.

15. The method of claim 14, wherein each set of mixed-type images consists of one monitor-mode image and one or more regular images.

16. The method of claim 14, wherein an activity metric is determined between one monitor-mode image and one previously captured image, and when the activity metric exceeds a threshold, one or more regular images are captured to form one set of mixed-types of images along with said one monitor-mode image.

17. The method of claim 16, wherein when the activity metric does not exceed the threshold, a next monitor-mode image is captured.

18. The method of claim 1, wherein the special images correspond to narrow-band images and the regular images correspond to wide-band images.

19. The method of claim 18, wherein each set of mixed-type images consists of one wide-band image and narrow-band images.

20. An apparatus for capturing an image sequence using a capsule camera when the capsule camera travels through a human gastrointestinal tract after the capsule camera is swallowed by a patient, wherein the capsule camera comprises an image sensor and is capable of capturing at least two mixed-type images corresponding to special images and regular images, the apparatus comprising one or more electronic circuits or processors arranged to:

determining one first energy-based frame time for the special images based on first light energy perceived by the image sensor;

determine one second energy-based frame time for the regular images based on second light energy perceived by the image sensor;

capture the image sequence comprising one or more sets of mixed-type images by configuring the capsule camera to cause a mixed-frame time distance between said one first energy-based frame time for one special image in a target set of mixed-type images and said one second energy-based frame time for one regular image in the target set of mixed-type images smaller than an overall average frame period, wherein each set of mixed-type images comprises at least one special image and one regular image; and

provide or output the image sequence captured for further processing or viewing; and

wherein the special images are used to control capturing the regular images or used to derive or provide associated information for the regular images; and

wherein an overall average frame rate is substantially lower than a capable frame rate of the capsule camera, and wherein the overall average frame rate is calculated as a sum of individual frame rates and each individual frame rate is associated with one type of said least two mixed-type images.