Stereo assist with rolling shutters

• US 10,200,638 B2
• Filed: 12/11/2017
• Issued: 02/05/2019
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Belman, Efim
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/838,344
• Publication Number:
  US 20180103218A1
• Time in Patent Office: 421 Days
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

An imaging system for a vehicle may include a first image capture device having a first field of view and configured to acquire a first image relative to a scene associated with the vehicle, the first image being acquired as a first series of image scan lines captured using a rolling shutter. The imaging system may also include a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, the second image capture device being configured to acquire a second image relative to the scene associated with the vehicle, the second image being acquired as a second series of image scan lines captured using a rolling shutter. As a result of overlap between the first field of view and the second field of view, a first overlap portion of the first image corresponds with a second overlap portion of the second image. The first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires the first overlap portion of the first image over a period of time during which the second overlap portion of the second image is acquired.

Claims

1. An imaging system, the system comprising:

a first image capture device having a first field of view and configured to acquire a first series of image scan lines relative to a scene;

a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, the second image capture device being configured to acquire a second series of image scan lines relative to the scene; and

at least one processing device configured to identify at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line and calculate a depth of at least one point based on the synchronized line.

2. The imaging system of claim 1, wherein:

the first image capture device has a first focal length;

the second image capture device has a second focal length greater than the first focal length; and

the at least one processing device is further configured to subsample the second series of image scan lines to reduce an effective focal length of the second image capture device to the first focal length.

3. The imaging system of claim 1, wherein the at least one processing device is further configured to identify synchronized points in one or more lines of the first series of image scan lines and the second series of images scan lines adjacent to the synchronized line, and the calculated depth is further based on the synchronized points.

4. The imaging system of claim 1, wherein the at least one processing device is further configured to calculate a disparity between points corresponding to the at least one point in the identified at least one scan line from the first series of images and the identified at least one scan line from the second series on images and to calculate the depth of the at least one point based on the disparity.

5. The imaging system of claim 4, wherein calculating the depth based on the disparity is further based on a stereo baseline of the system.

6. The imaging system of claim 1, wherein the at least one processing device is further configured to generate a depth map based on the calculated depth.

7. The imaging system of claim 6, wherein the depth map is based on calculated depths of scan lines above and below the synchronized line.

8. The imaging system of claim 1, wherein the at least one processing device is further configured to perform preprocessing on the first series of image scan lines and the second series of image scan lines based on homography of a target plane.

9. The imaging system of claim 1, wherein the first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires a first portion of the first series of image scan lines that overlaps a second portion of the second series of image scan lines over a period of time during which the second portion is acquired.

10. The imaging system of claim 9, wherein the synchronized line is selected from a subset of scan lines from the second image capture device within the second portion.

11. The imaging system of claim 1, the first series of image scan lines being captured using a first rolling shutter.

12. The imaging system of claim 11, the second series of image scan lines being captured using a second rolling shutter.

13. The imaging system of claim 12, wherein the at least one processing device is further configured to synchronize the first rolling shutter with the second rolling shutter.

14. An imaging system, comprising:

at least one processing device configured to;

receive, from a first image capture device having a first field of view, a first series of image scan lines relative to a scene;

receive, from a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, a second series of image scan lines relative to the scene; and

correlate at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line; and

calculate a depth of at least one point based on the synchronized line.

15. The imaging system of claim 14, wherein the at least one processing device is further configured to calculate a disparity between points corresponding to the at least one point in the identified at least one scan line from the first series of images and the identified at least one scan line from the second series on images and to calculate the depth of the at least one point based on the disparity.

16. The imaging system of claim 15, wherein calculating the depth based on the disparity is further based on a stereo baseline of the system.

17. The imaging system of claim 14, wherein the at least one processing device is further configured to identify synchronized points in one or more lines of the first series of image scan lines and the second series of images scan lines adjacent to the synchronized line, and the calculated depth is further based on the synchronized points.

18. The imaging system of claim 14, wherein the at least one processing device is further configured to generate a depth map based on the calculated depth.

19. The imaging system of claim 18, wherein the depth map is based on calculated depths of scan lines above and below the synchronized line.

20. The imaging system of claim 14, wherein the at least one processing device is further configured to perform preprocessing on the first series of image scan lines and the second series of images scan lines based on homography of a target plane.

21. The imaging system of claim 14, wherein the first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires a first portion of the first series of image scan lines that overlaps a second portion of the second series of image scan lines over a period of time during which the second portion is acquired.

22. The imaging system of claim 21, wherein the synchronized line is selected from a subset of scan lines from the second image capture device within the second portion.

23. The imaging system of claim 14, the first series of image scan lines being captured using a first rolling shutter.

24. The imaging system of claim 23, the second series of image scan lines being captured using a second rolling shutter.

25. The imaging system of claim 24, wherein the at least one processing device is further configured to perform a periodic check to determine whether the first rolling shutter and the second rolling shutter are synchronized.

26. The imaging system of claim 24, wherein the at least one processing device is further configured to perform resynchronization of the first rolling shutter and the second rolling shutter if the first rolling shutter and the second rolling shutter have fallen out of synch.

27. The imaging system of claim 14, wherein the first image capture device has a first focal length, the second image capture device has a second focal length greater than the first focal length, and the at least one processing device is further configured to subsample the second series of image scan lines to reduce an effective focal length of the second image capture device to the first focal length.

28. A method for depth computation, comprising:

receiving, from a first image capture device having a first field of view, a first series of image scan lines relative to a scene;

receiving, from a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, a second series of image scan lines relative to the scene; and

correlating at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line; and

calculating a depth of at least one point based on the synchronized line.