Increasing tolerance of sensor-scanner misalignment of the 3D camera with epipolar line laser point scanning

• US 9,661,308 B1
• Filed: 01/06/2016
• Issued: 05/23/2017
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ovsiannikov, Ilia, Deane, Peter, Wang, Yibing Michelle
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US14/989,716
• Publication Number:
  US 20170134710A1
• Time in Patent Office: 503 Days
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 25/745 Circuitry for generating timing or clock signals
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/78 Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters

Abstract

Using the same image sensor to capture a two-dimensional (2D) image and three-dimensional (3D) depth measurements for a 3D object. A laser point-scans the surface of the object with light spots, which are detected by a pixel array in the image sensor to generate the 3D depth profile of the object using triangulation. Each row of pixels in the pixel array forms an epipolar line of the corresponding laser scan line. Timestamping provides a correspondence between the pixel location of a captured light spot and the respective scan angle of the laser to remove any ambiguity in triangulation. An Analog-to-Digital Converter (ADC) in the image sensor operates as a Time-to-Digital Converter (TDC) to generate timestamps. When the epipolar line is misaligned or curved, multiple TDC arrays acquire timestamps of multiple pixels (in multiple rows) substantially simultaneously. Multiple timestamp values are reconciled to obtain a single timestamp value for a light spot.

Claims

1. A method comprising:

performing a one-dimensional (1D) point scan of a three-dimensional (3D) object along a scanning line using a light source, wherein the point scan projects a sequence of light spots on a surface of the 3D object;

selecting a plurality of rows of pixels in an image sensor, wherein the image sensor has a plurality of pixels arranged in a two-dimensional (2D) array forming an image plane, and wherein at least one of the plurality of the selected rows forms a portion of an epipolar line of the scanning line on the image plane;

for pixels in the selected row of pixels, sensing a pixel-specific detection of a corresponding light spot in the sequence of light spots;

for a group of pixels detecting a common light spot, generating a pixel-specific timestamp value for each pixel in the group substantially simultaneously; and

determining a distance to the common light spot on the surface of the 3D object based at least on the generated pixel-specific timestamp values and on a scan angle used by the light source for projecting the common light spot.

2. The method of claim 1, wherein at least two rows in the plurality of rows of pixels are adjacent rows.

3. The method of claim 1, wherein the 2D array is a Complementary Metal Oxide Semiconductor (CMOS) array.

4. The method of claim 1, wherein the 2D array is a Red Green Blue (RGB) array of pixels in which Red, Green, and Blue pixels are arranged in a Bayer pattern, and

wherein selecting the plurality of rows of pixels includes activating only Green pixels in the selected rows of pixels, and

wherein generating the pixel-specific timestamp value includes generating the pixel-specific timestamp value only for each activated Green pixel.

5. The method of claim 4, wherein pixels in the RGB array are arranged in a shared-pixel configuration resulting in a plurality of pairs of individually-selectable pixels of different colors, and wherein each pair of pixels includes one Green pixel.

6. The method of claim 1, wherein generating the pixel-specific timestamp value includes generating the pixel-specific timestamp value using a binary counter.

7. The method of claim 1, wherein determining the distance to the common light spot includes:

using all pixel-specific timestamp values associated with the group of pixels to obtain a group-specific timestamp value for the common light spot;

using the group-specific timestamp value for the common light spot to establish a correspondence between the following;

an offset distance between a collection optics associated with the image sensor and a pixel in the group of pixels whose pixel-specific timestamp value is closest to the group-specific timestamp value for the common light spot, and

the scan angle of the light source; and

determining the distance to the common light spot based on the offset distance and the scan angle.

8. The method of claim 7, wherein using all pixel-specific timestamp values includes:

for the group of pixels, selecting the pixel-specific timestamp value associated with a pixel that is physically substantially in a center of the group of pixels as the group-specific timestamp value for the common light spot.

9. The method of claim 7, further comprising:

for two or more groups of pixels in the selected rows of pixels, using group-specific timestamp values to establish a temporal correlation among respective time-stamped light spots; and

determining distances to the time-stamped light spots in an order specified by the temporal correlation.

10. The method of claim 1, wherein determining the distance to the common light spot includes:

for each pixel in the group of pixels, calculating a pixel-specific depth value; and

performing one of the following to determine the distance to the common light spot;

selecting the pixel-specific depth value associated with a pixel that is physically substantially in a center of the group of pixels as the distance to the common light spot, and

using a weighted average of all pixel-specific depth values associated with the group of pixels as the distance to the common light spot.

11. An imaging unit comprising:

a light source operative to perform a one-dimensional (1D) point scan of a three-dimensional (3D) object along a scanning line, wherein the point scan projects a sequence of light spots on a surface of the 3D object; and

an image sensor unit that includes;

a plurality of pixels arranged in a two-dimensional (2D) pixel array forming an image plane, wherein each pixel in a column of pixels in the 2D pixel array is associated with a respective row of pixels in the 2D pixel array, and wherein each pixel in the column of pixels is operative to detect a corresponding light spot in the sequence of light spots,

a plurality of Analog-to-Digital Converter (ADC) units per column of pixels in the 2D pixel array, wherein at least two adjacent pixels in a column of pixels are connected to different ADC units, and wherein each ADC unit is operative to generate a pixel-specific timestamp value for a respective pixel connected thereto in response to a pixel-specific detection of the corresponding light spot by the respective pixel, and

a processing unit coupled to the plurality of ADC units, wherein the processing unit is operative to perform the following;

select a plurality of rows of pixels in the 2D pixel array associated with a group of pixels detecting a common light spot in the sequence of light spots, wherein at least one of the plurality of the selected rows forms a portion of an epipolar line of the scanning line on the image plane,

for pixels in the selected rows of pixels, sense a pixel-specific detection of the common light spot,

trigger the ADC units associated with the pixels in the group of pixels so as to substantially simultaneously generate a pixel-specific timestamp value for each pixel in the group of pixels, and

determine a distance to the common light spot on the surface of the 3D object based at least on the generated pixel-specific timestamp values and on a scan angle used by the light source for projecting the common light spot.

12. The imaging unit of claim 11, wherein the 2D pixel array is a Red Green Blue (RGB) array of pixels in which Red, Green, and Blue pixels are arranged in a shared-pixel configuration resulting in a plurality of pairs of individually-selectable pixels of different colors, wherein each pair of pixels includes one Green pixel, and wherein two consecutive Green pixels in a column are connected to two different ADC units,

and wherein the processing unit is operative to further perform the following:

activate only Green pixels in the group of pixels,

sense the pixel-specific detection of the common light spot only for each activated Green pixel, and

trigger the ADC units associated only with the Green pixels in the group of pixels so as to substantially simultaneously generate the pixel-specific timestamp value only for each activated Green pixel.

13. The imaging unit of claim 11, wherein the processing unit is operative to perform the following to determine the distance to the common light spot:

for the group of pixels, select only one of the generated pixel-specific timestamp values as a group-specific timestamp value for the common light spot;

use the group-specific timestamp value for the common light spot to establish a correspondence between the following;

an offset distance between a collection optics associated with the image sensor unit and a pixel in the group of pixels whose pixel-specific timestamp value is selected as the group-specific timestamp value for the common light spot, and

the scan angle of the light source; and

determine the distance to the common light spot based on the offset distance and the scan angle.

14. A system comprising:

a light source operative to perform a one-dimensional (1D) point scan of a three-dimensional (3D) object along a scanning line, wherein the point scan projects a sequence of light spots on a surface of the 3D object;

a plurality of pixels arranged in a two-dimensional (2D) pixel array forming an image plane, wherein each pixel in a column of pixels in the 2D pixel array is associated with a respective row of pixels in the 2D pixel array, and wherein each pixel in the column of pixels is operative to detect a corresponding light spot in the sequence of light spots;

a plurality of Analog-to-Digital Converter (ADC) units per column of pixels in the 2D pixel array, wherein at least two adjacent pixels in a column of pixels are connected to different ADC units, and wherein each ADC unit is operative to generate a pixel-specific timestamp value for a respective pixel connected thereto in response to a pixel-specific detection of the corresponding light spot by the respective pixel;

a memory for storing program instructions; and

a processor coupled to the memory and to the plurality of ADC units, wherein the processor is configured to execute the program instructions, whereby the processor is operative to perform the following;

select a plurality of rows of pixels in the 2D pixel array associated with a group of pixels detecting a common light spot in the sequence of light spots, wherein at least one of the plurality of the selected rows forms a portion of an epipolar line of the scanning line on the image plane,

for pixels in the selected rows of pixels, sense a pixel-specific detection of the common light spot,

trigger the ADC units associated with the pixels in the group of pixels so as to substantially simultaneously generate a pixel-specific timestamp value for each pixel in the group of pixels, and

determine a distance to the common light spot on the surface of the 3D object based at least on the generated pixel-specific timestamp values and on a scan angle used by the light source for projecting the common light spot.

15. The system of claim 14, wherein the 2D pixel array is a Complementary Metal Oxide Semiconductor (CMOS) array, and wherein each pixel in the row of pixels is one of the following:

a Four Transistor (4T) pixel;

a Three Transistor (3T) pixel;

a Two Transistor (2T) pixel;

a One Transistor (1T) pixel;

a shared-transistor pixel;

a 1×

2 2-shared pixel; and

a 1×

4 4-shared pixel.

16. The system of claim 14, wherein the light source is one of the following:

a laser light source;

a visible light source;

a point light source;

a near infrared laser light source;

a monochromatic illumination source; and

an X-Y addressable light source.

17. The system of claim 14, wherein, upon execution of the program instructions, the processor is operative to perform the following to determine the distance to the common light spot:

for each pixel in the group of pixels, calculate a pixel-specific depth value; and

perform one of the following to determine the distance to the common light spot;

select the pixel-specific depth value associated with a pixel that is physically substantially in a center of the group of pixels as the distance to the common light spot, and

use a weighted average of all pixel-specific depth values associated with the group of pixels as the distance to the common light spot.

18. The system of claim 14, wherein adjacent pixels in each selected row of pixels in the 2D pixel array have different colors, and

wherein, upon execution of the program instructions, the processor is operative to further perform the following:

activate only those pixels in the selected rows of pixels that have the same color, and

trigger the ADC units associated only with the activated pixels in the group of pixels so as to substantially simultaneously generate the pixel-specific timestamp value only for each activated pixel.

19. The system of claim 14, wherein, upon execution of the program instructions, the processor is operative to perform the following to determine the distance to the common light spot:

use all pixel-specific timestamp values associated with the group of pixels to obtain a group-specific timestamp value for the common light spot;

use the group-specific timestamp value for the common light spot to establish a correspondence between the following;

an offset distance between a collection optics associated with the 2D pixel array and a pixel in the group of pixels whose pixel-specific timestamp value is closest to the group-specific timestamp value for the common light spot, and

the scan angle of the light source; and

determine the distance to the common light spot based on the offset distance and the scan angle.

20. The system of claim 14, wherein the system is one of the following:

a mobile device;

a smartphone;

a tablet computer;

a computing unit;

a User Equipment (UE);

an automobile;

a virtual reality equipment; and

a robot.