Method and apparatus for endoscope with distance measuring for object scaling

• US 10,402,992 B2
• Filed: 01/30/2018
• Issued: 09/03/2019
• Est. Priority Date: 10/16/2015
• Status: Active Grant

• Current Assignee: CapsoVision, Inc.
• Original Assignee: CapsoVision, Inc.
• Inventors: Wang, Kang-Huai, Wu, Chenyu, Xu, Yi
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/883,337
• Publication Number:
  US 20180174318A1
• Time in Patent Office: 581 Days
• Field of Search: 348 65
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/041 Capsule endoscopes for imaging
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: A61B 1/0661 Endoscope light sources
• CPC Class Codes: A61B 5/073 Intestinal transmitters
• CPC Class Codes: A61B 5/1076 for measuring dimensions inside body cavities, e.g. using catheters A61B3/1005 takes precedence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30028 Colon; Small intestine
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A method and apparatus for capturing images of a scene using a capsule device including a camera are disclosed. An image sequence is captured using the camera when the capsule device travels through a human gastrointestinal tract. Also, structured-light images are captured using the camera by projecting structured light to one or more objects in a field of view of the camera when the capsule device travels through the human gastrointestinal tract. The structured-light images are interleaved with regular images in the image sequence. The distance information with respect to the capsule camera associated with objects of the selected image is derived. Both the image sequence and the distance information are outputted. A method of determining the size of an object of interest utilizing the distance information is also disclosed. In another method, the distance information is used to scale object or adjust intensities.

Claims

1. A method of capturing images of a scene using a capsule device including a camera, the method comprising:

administering the capsule device to a patient;

projecting non-structured light from a non-structured light source to a scene in a field of view of the camera;

capturing, by the camera, regular images formed on a common image plane of the camera when the capsule device travels through a human gastrointestinal tract;

projecting structured light from a structured light source to the scene in the field of view of the camera;

capturing, by the camera, structured-light images formed on the common image plane of the camera when the capsule device travels through the human gastrointestinal tract, wherein the structured-light images are interleaved with the regular images to derive distances associated with objects in the regular images with respect to the camera; and

outputting the regular images and

outputting the structured-light images for deriving information comprising the distances associated with the objects in the regular images based on the structured-light images or outputting derived information comprising the distances associated with the objects in the regular images, wherein each of the distances corresponds to one derived distance between one object in the regular image and the camera.

2. The method of claim 1 further comprises deriving the distances associated with the objects in the regular images based on the structured-light images.

3. The method of claim 2, wherein the derived information further comprises association information between the distances and corresponding images of the regular images.

4. The method of claim 3, wherein the association information corresponds to frame numbers or capture times of the corresponding images of the regular images.

5. An endoscope for capturing images of a scene using a capsule device, the capsule device comprising:

a camera;

a non-structured light source;

a structured light source;

one or more processors coupled to the camera and the structured light source;

one or more output interfaces coupled to said one or more processors; and

wherein said one or more processors are configured to;

capture regular images using the camera by projecting non-structured light from the non-structured light source when the capsule device travels through a human gastrointestinal tract;

capture structured-light images using the camera by projecting structured light from the structured light source to a scene in a field of view of the camera when the capsule device travels through the human gastrointestinal tract, wherein the structured-light images are interleaved with the regular images to derive distances associated with objects in the regular images with respect to the camera;

output the regular images through said one or more output interfaces; and

output the structured-light images for deriving information comprising the distances associated with the objects in the regular images based on the structured-light images or outputting derived information comprising the distances associated with the objects in the regular images, wherein each of the distances corresponds to one derived distance between one object in the regular image and the camera; and

a housing adapted to be swallowed, wherein the housing encloses the camera and said one or more processors in a sealed environment.

6. The capsule device of claim 5, wherein said one or more processors are configured further to derive the distances associated with the objects in the regular images based on the structured-light images.

7. The capsule device of claim 5, wherein the derived information further comprises association information between the distances and corresponding images of the regular images.

8. The capsule device of claim 7, wherein the association information corresponds to frame numbers or capture times of the corresponding images of the regular images.

9. A method of processing images captured using a capsule device including a camera, the method comprising:

receiving regular images captured by the camera when the capsule device traveled through a human gastrointestinal tract;

determining information comprising a distance associated with a target object in a selected regular image based on one or more structured-light images captured by the camera when the capsule device traveled through the human gastrointestinal tract, wherein the distance corresponds to one derived distance between the target object in the selected regular image and the camera;

determining a physical size of the target object in the selected regular image based on pixel data of the target object and the distance between the target object in the selected regular image and the camera; and

outputting size information regarding the physical size of the target object.

10. The method of claim 9, wherein the physical size of the target object is determined according to an image size of the target object in the selected regular image scaled by a ratio of distance associated with the target object to the camera and focal length of the camera.

11. The method of claim 10, wherein the image size of the target object is measured in terms of a number of pixels of the target object in the selected regular image.

12. The method of claim 9 further comprising receiving the structured-light images from the camera and deriving the distance associated with the target object in the regular images based on the structured-light images.

13. A method of processing images captured using a capsule device including a camera, the method comprising:

receiving regular images captured by the camera when the capsule device traveled through a human gastrointestinal tract;

determining information comprising a distance associated with a target object in a selected regular image based on one or more structured-light images captured by the camera when the capsule device traveled through the human gastrointestinal tract, wherein the distance corresponds to one derived distance between the target object in the selected regular image and the camera;

stitching the regular images utilizing the information including the distance to generate a stitched image sequence; and

outputting the stitched image sequence.

14. The method of claim 13 further comprising receiving the structured-light images from the camera and deriving the distance corresponding to one derived distance between the target object in the selected regular image and the camera based on the structured-light images.

15. The method of claim 14, wherein the distance is used to scale the target objects in the regular images for said stitching the regular images.

16. The method of claim 14, wherein the distance is used to adjust image intensities of at least one regular image for said stitching the regular images.

17. The method of claim 13, wherein association information between the distance and corresponding images of the regular images is also received and used for said stitching the regular images.

18. An apparatus for processing images captured using a capsule device including a camera, the apparatus comprising one or more electronic circuits or processors arranged to:

receive regular images captured by the camera when the device traveled through a human gastrointestinal tract;

determine information comprising a distance associated with a target object in a selected regular image based on one or more structured-light images captured by the camera when the capsule device traveled through the human gastrointestinal tract, wherein the distance corresponds to one derived distance between the target object in the selected regular image and the camera;

stitch the regular images utilizing the information including the distance to generate a stitched image sequence; and

output the stitched image sequence.

19. The apparatus of claim 18, wherein said one or more electronic circuits or processors are arranged further to receive the structured-light images from the camera and derive the distance corresponding to one derived distance between the target object in the selected regular image and the camera based on the structured-light images.

20. The apparatus of claim 18, wherein association information between the distance and corresponding images of the regular images is also received and used for stitching the regular images.