Building night vision and other driver assistance systems (DAS) using near infra-red (NIR) illumination and rolling shutter

• US 10,880,471 B2
• Filed: 11/12/2018
• Issued: 12/29/2020
• Est. Priority Date: 10/31/2010
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Belman, Efim
• Primary Examiner(s): Huang, Frank F
• Application Number:
  US16/350,373
• Publication Number:
  US 20190109987A1
• Time in Patent Office: 778 Days
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A system mountable in a motor vehicle. The system includes a camera and a processor configured to receive image data from the camera. The camera includes a rolling shutter configured to capture the image data during a frame period and to scan and to read the image data into multiple image frames. A near infra-red illuminator may be configured to provide a near infra-red illumination cone in the field of view of the camera. The near infra-red illumination oscillates with an illumination period. A synchronization mechanism may be configured to synchronize the illumination period to the frame period of the rolling shutter. The frame period may be selected so that the synchronization mechanism provides a spatial profile of the near infra-red illumination cone which may be substantially aligned vertically to a specific region, e.g. near the center of the image frame.

Claims

1. A system comprising:

a processor configured to;

receive image data from a camera, the camera configured to scan the image data line-by-line into a plurality of image frames;

generate a plurality of infra-red image frames, the plurality of infra-red image frames including infra-red image data scanned while an illuminator provides a first intensity of infra-red illumination in a field of view of the camera; and

generate a plurality of color image frames, the plurality of color image frames including color image data scanned while the illuminator provides the second intensity of infra-red illumination, the first intensity of infra-red illumination greater than the second intensity of infra-red illumination;

wherein;

the first and second infra-red illuminations oscillate with an oscillation frequency and an illumination duty cycle; and

the infra-red illuminator provides the first and second infra-red illuminations based on a synchronization signal provided by a synchronization mechanism configured to synchronize the oscillation frequency and the illumination duty cycle line-by-line with an exposure of the image frames and vary the intensity of the first and second infra-red illuminations between at least two lines during line-by-line scanning of image data by the camera.

2. The system according to claim 1, wherein:

the infra-red illuminator provides near infra-red illumination; and

the plurality of infra-red image frames includes near infra-red image data.

3. The system according to claim 1, wherein:

the plurality of image frames include a long exposure frame and a short exposure frame; and

the synchronization mechanism is configured to initiate the activation of the infra-red illumination during exposure of at least one row from a central region but not all rows of the long exposure frame.

4. The system according to claim 1, wherein:

the camera includes a rolling shutter configured to capture the image data during a frame period; and

the frame period of the camera is selected so that the synchronization mechanism provides a spatial profile of an infra-red illumination cone substantially aligned vertically to a specific region of at least a portion of the image frames.

5. The system according to claim 1, wherein:

the processor includes a plurality of applications;

a first application of the plurality of applications is configured to use the plurality of infra-red image frames; and

a second application of the plurality of applications, different from the first application, is configured to use the plurality of color image frames.

6. The system according to claim 5, wherein the plurality of image frames is partitioned into first image frames for use by the first application and into second image frames for use by the second application.

7. The system according to claim 6, wherein the synchronization module is configured to:

time the first intensity of infra-red illumination to be in at least a portion in each of the plurality of infra-red image frames; and

time the second intensity of infra-red illumination to be in at least a portion in each of the color image frames.

8. The system according to claim 6, wherein a parameter of the camera is configurable to alternate respective values between the first and second image frames.

9. The system according to claim 8, wherein the parameter is a frame exposure time of the camera.

10. The system according to claim 6, wherein the first application is configured to perform detection of a person responsive to the infra-red illumination in the first image frames.

11. The system according to claim 6, wherein the second application uses color information from the second image frames to detect one or more types of light sources in the field of view of the camera.

12. The system according to claim 6, wherein the second application uses color information from the second image frames to detect a sign in the field of view of the camera.

13. The system according to claim 1, wherein:

the synchronization mechanism is configured to turn off the infra-red illumination during exposure of at least one middle row of an image frame; and

an application of the processor uses color information from the image frame.

14. The system according to claim 1, wherein:

the synchronization mechanism is configured to turn off the infra-red illumination during exposure of at least one middle row of an image frame; and

an application of the processor uses the image frame to recognize reflective signs.

15. The system according to claim 1, wherein the synchronization mechanism is configured to turn on the infra-red illumination for less than an image frame capture duration of the plurality of image frames.

16. The system according to claim 1, wherein:

the second intensity is provided while the first line of the first image frame of the plurality of image frames is scanned; and

the first intensity is substantially zero.

17. A method comprising:

receiving image data at a processor from a camera, the camera configured to scan the image data line-by-line into a plurality of image frames;

generating a plurality of infra-red image frames at the processor, the plurality of infra-red image frames including infra-red image data scanned while an illuminator provides a first intensity of infra-red illumination in a field of view of the camera; and

generate a plurality of color image frames at the processor, the plurality of color image frames including non infra-red image data scanned while the illuminator provides the second intensity of infra-red illumination, the first intensity of infra-red illumination greater than the second intensity of infra-red illumination;

wherein;

the first and second infra-red illuminations oscillate with an oscillation frequency and an illumination duty cycle; and

the infra-red illuminator provides the first and second infra-red illuminations based on a synchronization signal provided by a synchronization mechanism configured to synchronize the oscillation frequency and the illumination duty cycle line-by-line with an exposure of the image frames and vary the intensity of the first and second infra-red illuminations between at least two lines during line-by-line scanning of image data by the camera.

18. The method according to claim 17, wherein:

the infra-red illuminator provides near infra-red illumination; and

the plurality of infra-red image frames includes near infra-red image data.

19. The method according to claim 17, wherein:

the plurality of image frames includes a long exposure frame and a short exposure frame; and

the synchronization initiates the activation of the infra-red illumination during exposure of at least one row from a central region but not all rows of the long exposure frame.

20. The method according to claim 17, wherein

the camera includes a rolling shutter configured to capture the image data during a frame period;

and

the frame period of the camera is selected so that the synchronization provides a spatial profile of an infra-red illumination cone substantially aligned vertically to a specific region of at least a portion of the image frames.

21. The method according to claim 17, wherein:

the processor includes a plurality of applications;

a first application of the plurality of applications is configured to use the plurality of infra-red image frames; and

a second application of the plurality of applications, different from the first application, is configured to use the plurality of color image frames.

22. The method according to claim 21, further comprising configuring the camera to provide to the processor the plurality of image frames, wherein the plurality of image frames is partitioned into first image frames for use by the first application and into second image frames for use by the second application.

23. The method according to claim 22, wherein the synchronization provides the first intensity of infra-red illumination in at least a portion in each of the infra-red image frames and provides the second intensity of infra-red illumination in at least a portion in each of the color image frames.

24. The method according to claim 22, further comprising alternating respective values of a camera parameter in alternating first and second image frames.

25. The method according to claim 24, wherein the camera parameter is a frame exposure time of the camera.

26. The method according to claim 22, wherein the first application is configured to perform detection of a person responsive to the infra-red illumination in the first image frames.

27. The method according to claim 22, wherein the second application uses color information from the second image frames to detect one or more types of light sources in the field of view of the camera.

28. The method according to claim 22, wherein the second application uses color information from the second image frames to detect a sign in the field of view of the camera.

29. The method according to claim 17, wherein:

the second intensity is provided while the first line of the first image frame of the plurality of image frames is scanned; and

the first intensity is substantially zero.

30. The method according to claim 17, wherein:

the synchronization turns off the infra-red illumination during exposure of at least one middle row of an image frame; and

an application of the processor uses color information from the image frame.

31. The method according to claim 17, wherein:

the synchronization turns off the infra-red illumination during exposure of at least one middle row of an image frame; and

an application of the processor uses the image frame to recognize reflective signs.

32. The method according to claim 17, wherein the synchronization turns on the infra-red illumination for less than an image frame capture duration of the plurality of image frames.