Method and apparatus for a computer vision camera unit

• US 10,484,666 B1
• Filed: 03/17/2017
• Issued: 11/19/2019
• Est. Priority Date: 03/17/2016
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, Edge Robotics, Inc.
• Original Assignee: Herman Herman
• Inventors: Herman, Herman
• Primary Examiner(s): Perkey, William B
• Application Number:
  US15/462,707
• Time in Patent Office: 977 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H01L 27/14601 Structural or functional details thereof
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A computer vision camera unit comprising first and second pairs, of image capture assemblies, both of which have an approximately same orientation to a first axis. The first and second pairs are optically focused to capture first and second fields of view that, at least, overlap each other. Providing the first and second pairs with imager chips of differing light sensitivity permits high dynamic range to be realized. Having the second field of view within the first field of view permits specialization, between the first and second pairs, similar to the roles of peripheral and foveal vision, found in biological stereo vision systems. The image capture assemblies can be organized in a rectangular configuration that implies third and fourth pairs, orthogonal to the first and second pairs. The orthogonal but interlocking nature, of the first and second pairs in relation to the third and fourth pairs, enables an autocalibration technique.

Claims

1. A method for a first computer vision camera unit, comprising:

identifying a same first feature with first, second, third, and fourth image capture assemblies, each image capture assembly having a lens assembly embedded, in a first frame, at a vertex of a first rectangular configuration;

identifying, with first, second, third, and fourth imager chips, of, respectively, the first, second, third, and fourth image capture assemblies, the same first feature as being at first, second, third, and fourth locations, relative to, respectively, first, second, third, and fourth principal points;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if, with respect to a first axis, there is a first misalignment between the first and second locations;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if there is the first misalignment, a first change, for each of the first and second principal points, that reduces the misalignment to zero;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if, with respect to the first axis, there is second misalignment between the third and fourth locations;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if there is a second misalignment, a second change, for each of the third and fourth principal points, that reduces the misalignment to zero;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if, with respect to a second axis, there is third misalignment between the first and third locations;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if there is a third misalignment, a third change, for each of the first and third principal points, that reduces the misalignment to zero;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if, with respect to the second axis, there is a fourth misalignment between the second and fourth locations;

determining, performed at least in part with a configuration of computing hardware and programmable memory, if there is a fourth misalignment, a fourth change, for each of the second and fourth principal points, that reduces the misalignment to zero;

determining, performed at least in part with a configuration of computing hardware and programmable memory, as a result of the first and third changes, a first distance by which the first principal point changes its location;

determining, performed at least in part with a configuration of computing hardware and programmable memory, as a result of the first and fourth changes, a second distance by which the second principal point changes its location;

determining, performed at least in part with a configuration of computing hardware and programmable memory, as a result of the second and third changes, a third distance by which the third principal point changes its location;

determining, performed at least in part with a configuration of computing hardware and programmable memory, as a result of the second and fourth changes, a fourth distance by which the fourth principal point changes its location;

determining, performed at least in part with a configuration of computing hardware and programmable memory, the first, second, third, and fourth changes such that a sum, of first, second, third, and fourth distances, is minimized;

determining, performed at least in part with a configuration of computing hardware and programmable memory, a first depth, of the first feature from the first computer vision camera unit, based upon a change of location of the first feature, along the second axis, between the first and second locations;

determining, performed at least in part with a configuration of computing hardware and programmable memory, a second depth, of the first feature from the first computer vision camera unit, based upon a change of location of the first feature, along the first axis, between the first and third locations; and

rejecting, performed at least in part with a configuration of computing hardware and programmable memory, a set, of first, second, third, and fourth changes, if the first depth does not equal the second depth.

8. The method of claim 1, wherein the first, second, third, and fourth changes are determined, for minimization of their sum, by a non-heuristic iterative method.

9. The method of claim 1, wherein the first, second, third, and fourth changes are determined, for minimization of their sum, by an at least partially stochastic method.

2. A method for a first computer vision camera unit, comprising:

capturing a first pair of images, for use with stereo depth perception, from a first pair of image capture assemblies oriented according to a first axis and optically focused to capture a first field of view;

capturing a second pair of images, for use with stereo depth perception, from a second pair of image capture assemblies oriented according to the first axis and optically focused to capture a second field of view;

wherein the first and second fields of view are optically focused to overlap;

wherein the first and second fields of view are optically focused such that the second field of view is within the first field of view;

capturing the first pair of images with a first pair of imager chips with a first density of photo-sensitive elements;

capturing the second pair of images with a second pair of imager chips with a second density of photo-sensitive elements; and

wherein the first and second densities are different and are selected such that a second resolution, of the second pair of images, is greater than a first resolution of the first pair of images.

10. The method of claim 2, wherein the first and second pairs of image capture assemblies form a rectangular configuration.

3. A method for a first computer vision camera unit comprising:

capturing a first pair of images, for use with stereo depth perception, from a first pair of image capture assemblies oriented according to a first axis and optically focused to capture a first field of view;

capturing a second pair of images, for use with stereo depth perception, from a second pair of image capture assemblies oriented according to the first axis and optically focused to capture a second field of view;

wherein the first and second fields of view are optically focused to overlap;

wherein the first and second fields of view are optically focused such that the second field of view is smaller than the first field of view;

capturing the first pair of images with a first pair of imager chips;

capturing the second pair of images with a second pair of imager chips;

capturing at a first rate, with the first pair of imager chips, a first series of pairs of images;

capturing at a second rate, with the second pair of imager chips, a second series of pairs of images; and

wherein the first rate is higher than the second rate.

4. The method of claim 3, wherein the first rate is selected to better capture motion while the second rate is chosen to better capture detail.

5. The method of claim 3, wherein the first series, of pairs of images, is captured as monochrome, while the second series, of pairs of images, includes a capture of color.

11. The method of claim 3, wherein the first and second pairs of image capture assemblies form a rectangular configuration.

6. A method for a first computer vision camera unit, comprising:

capturing a first pair of images, for use with stereo depth perception, from a first pair of image capture assemblies oriented according to a first axis and optically focused to capture a first field of view;

capturing a second pair of images, for use with stereo depth perception, from a second pair of image capture assemblies oriented according to the first axis and optically focused to capture a second field of view;

wherein the first and second fields of view are optically focused to overlap;

capturing the first pair of images with a first pair of imager chips with a first level of light sensitivity for its photo-sensitive elements;

capturing the second pair of images with a second pair of imager chips with a second level of light sensitivity for its photo-sensitive elements; and

wherein the first and second light sensitivity levels are selected to be substantially different from each other.

7. The method of claim 6, wherein the first and second light sensitivity levels are selected to provide the first computer vision camera unit with a high dynamic range.

12. The method of claim 6, wherein the first and second pairs of image capture assemblies form a rectangular configuration.