Single image sensor for capturing mixed structured-light images and regular images

• US 10,484,629 B2
• Filed: 10/24/2016
• Issued: 11/19/2019
• Est. Priority Date: 10/16/2015
• Status: Active Grant

• Current Assignee: CapsoVision, Inc.
• Original Assignee: CapsoVision, Inc.
• Inventors: Wang, Kang-Huai, Xu, Yi, Wilson, Gordon C., Wu, Chenyu
• Primary Examiner(s): Chio, Tat C
• Assistant Examiner(s): Nguyen, Kathleen V
• Application Number:
  US15/333,071
• Publication Number:
  US 20170109890A1
• Time in Patent Office: 1,121 Days
• Field of Search: 348 46
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures

Abstract

A method and device for improving the accuracy of depth information derived from a structured-light image for a regular image are disclosed. In one example, an additional structured-light image is captured before a first structured-light image or after a regular image. The depth information for the regular image can be derived from the first structured-light image and corrected by incorporating depth information from the additional structured-light image. A model for depth information can be used to predict or interpolate depth information for the regular image. In another example, two regular sub-images may be captured with a structured-light image in between. If substantial frame differences or substantial global motion vector/block motion vectors are detected, the two regular sub-images will not be combined in order to avoid possible motion smear. Instead, one of the two sub-images will be selected and scaled as the output regular image.

Claims

1. A method of deriving depth information for a regular image from structured-light images using a camera comprising an image sensor, the method comprising:

projecting a first structured light to a scene in a field of view of the image sensor, wherein the image sensor comprises a pixel array;

applying first reset signals to the pixel array to reset rows of the pixel array;

exposing the rows of the pixel array to the first structured light to cause first analog signals from the rows of the pixel array;

capturing a first structured-light image using the pixel array during a first frame period by converting the first analog signals from the rows of the pixel array into first digital outputs to form the first structured-light image using one or more analog-to-digital converters;

projecting a second structured light to the scene in the field of view of the image sensor;

applying second reset signals to the pixel array to reset rows of the pixel array;

exposing the rows of the pixel array to the second structured light to cause second analog signals from the rows of the pixel array;

capturing a second structured-light image using the pixel array during a second frame period by converting the second analog signals from the rows of the pixel array into second digital outputs to form the second structured-light image using said one or more analog-to-digital converters;

applying third reset signals to the pixel array to reset the rows of the pixel array;

exposing the rows of the pixel array to non-structured light to cause third analog signals from the rows of the pixel array;

capturing the regular image using the pixel array during a third frame period by converting the third analog signals from the rows of the pixel array into third digital outputs to form the regular image using said one or more analog-to-digital converters;

deriving first depth information from the first structured-light image;

deriving second depth information from the second structured-light image; and

deriving current depth information associated with the regular image based on the first depth information and the second depth information; and

wherein same pixels of the pixel array are used for capturing the first structured-light image, the second structured-light image and the regular image;

wherein the regular image is captured before, after or between the first structured-light image and the second structured-light image; and

wherein the third frame period is longer than the first frame period and the second frame period.

2. The method of claim 1, wherein when the regular image is captured between the first structured-light image and the second structured-light image, the current depth information is derived based on the first depth information and differences between the first depth information and the second depth information, or based on the second depth information and the differences between the first depth information and the second depth information.

3. The method of claim 1, wherein the current depth information is derived row by row based on the first depth information and differences between the first depth information and the second depth information by taking into account of time elapsed between each corresponding rows in the first structured-light image and the regular image respectively, or based on the second depth information and the differences between the first depth information and the second depth information by taking into account of time elapsed between each corresponding rows in the second structured-light image and the regular image respectively.

4. The method of claim 1, wherein a depth model is derived based on the first depth information and the second depth information, and the current depth information is derived using the depth model.

5. The method of claim 4, wherein when the regular image is captured between the first structured-light image and the second structured-light image, the current depth information is interpolated using the depth model.

6. The method of claim 4, wherein when the regular image is captured after the first structured-light image and the second structured-light image, the current depth information is predicted using the depth model.

7. A capsule camera system, comprising:

an image sensor comprising a pixel array being responsive to light energy received by the image sensor to produce pixel signals having a voltage level depending on the light energy received;

a structured light source;

a non-structured light source;

one or more output circuits coupled to the image sensor to access the pixel signals produced by the image sensor;

one or more analog-to-digital converters having a first dynamic range and a second dynamic range;

one or more timing and control circuits, wherein said one or more timing and control circuits are arranged to;

project a first structured-light to a scene in a field of view of the image sensor;

apply first reset signals to the pixel array to reset rows of the pixel array;

expose the rows of the pixel array to the first structured light to cause first analog signals from the rows of the pixel array;

capture a first structured-light image using the image sensor during a first frame period by converting the first analog signals from the rows of the pixel array into first digital outputs to form the first structured-light image using one or more analog-to-digital converters;

project a second structured light to the scene in the field of view of the image sensor;

apply second reset signals to the pixel array to reset rows of the pixel array;

expose the rows of the pixel array to the second structured light to cause second analog signals from the rows of the pixel array;

capture a second structured-light image using the image sensor during a second frame period by converting the second analog signals from the rows of the pixel array into second digital outputs to form the second structured-light image using said one or more analog-to-digital converters;

apply third reset signals to the pixel array to reset the rows of the pixel array;

exposing the rows of the pixel array to non-structured light to cause third analog signals from the rows of the pixel array, and

capture a regular image using the image sensor during a third frame period by converting the third analog signals from the rows of the pixel array into third digital outputs to form the regular image using said one or more analog-to-digital converters;

derive first depth information from the first structured-light image;

derive second depth information from the second structured-light image; and

derive current depth information associated with the regular image based on the first depth information and the second depth information; and

wherein same pixels of the pixel array are used for the first structured-light image, the second structured-light image and the regular image;

wherein the regular image is captured before, after or between the first structured-light image and the second structured-light image; and

wherein the third frame period is longer than the first frame period and the second frame period.