Image data acquisition logic of an autonomous driving vehicle for capturing image data using cameras

• US 10,816,979 B2
• Filed: 08/24/2018
• Issued: 10/27/2020
• Est. Priority Date: 08/24/2018
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baidu USA LLC (Baidu Incorporated)
• Inventors: Zhang, Manjiang, You, Xiangtao, Zhang, Tiffany
• Primary Examiner(s): Yenke, Brian P
• Application Number:
  US16/112,370
• Publication Number:
  US 20200064837A1
• Time in Patent Office: 795 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/01 Conversion of standards , e.g. involving analogue television standards or digital television standards processed at pixel level

Abstract

A sensor unit used in an ADV includes a sensor interface that can be coupled to cameras mounted on the ADV. The sensor unit further includes a host interface that can be coupled to a host system. The host system is configured to perceive a driving environment surrounding the ADV based on at least image data obtained from the cameras and to plan a path to autonomously drive the ADV. The sensor unit further includes one or more data acquisition modules corresponding to the cameras. Each data acquisition module includes a pixel alignment module and a frame processing module. The pixel alignment module is configured to reformat pixels of image data from an original format associated with the corresponding camera to a predetermined format. The frame processing module is configured to generate an image frame based on the image data and to transmit the image frame to the host system.

Claims

1. A sensor unit utilized in an autonomous driving vehicle, the sensor unit comprising:

a sensor interface to be coupled to a plurality of cameras mounted on a plurality of locations of an autonomous driving vehicle (ADV);

a host interface to be coupled to a host system, wherein the host system is configured to perceive a driving environment surrounding the ADV based on at least image data obtained from the cameras and to plan a path to autonomously drive the ADV; and

a plurality of data acquisition processors, each corresponding to one of the plurality of cameras, wherein each of the data acquisition processors comprises;

a pixel alignment processor to reformat pixels of image data from an original format associated with a corresponding camera to a predetermined format, and

a frame processing processor coupled to the pixel alignment processor, wherein the frame processing processor is configured to generate an image frame based on the image data received from the pixel alignment processor and to transmit the image frame to the host system.

2. The sensor unit of claim 1, wherein each data acquisition processor further comprises a camera simulator processor coupled to the pixel alignment processor, wherein the camera simulator processor is configured to simulate a camera of the plurality of cameras to generate simulated image data without using the corresponding camera.

3. The sensor unit of claim 2, wherein the simulated image data is utilized for testing a functionality of the pixel alignment processor and the frame processing processor without having to couple to a camera.

4. The sensor unit of claim 1, wherein each data acquisition processor further comprises a host simulator processor coupled to the frame processing processor, wherein the host simulator processor is configured to simulate the host system by generating simulated host commands and responses without using the host system.

5. The sensor unit of claim 4, wherein the simulated host commands and responses are utilized to test a functionality of the pixel alignment processor and the frame processing processor without having to couple with the host system.

6. The sensor unit of claim 1, wherein the pixel alignment processor further comprises a pixel error detector processor to examine image data to detect a pixel error.

7. The sensor unit of claim 6, wherein the pixel error detector processor is configured to detect whether there is a sufficient number of pixels in the image data or whether the image data violates a camera transmission protocol.

8. The sensor unit of claim 1, wherein each data acquisition processor further comprises a frame error detector processor to examine the image frame to detect a frame error.

9. The sensor unit of claim 8, wherein the frame error detector processor is configured to detect whether the image frame is a short frame or a long frame.

10. The sensor unit of claim 1, wherein each data acquisition processor further comprises a trigger counter processor to store a trigger count counting a number of trigger signals that has been sent to the corresponding camera to capture a new image.

11. The sensor unit of claim 10, wherein the trigger count is included in an image frame that was captured in response to a latest trigger signal.

12. The sensor unit of claim 1, wherein each data acquisition processor further comprises a timestamp generator processor to generate a timestamp recording time at which the image data is received from the corresponding camera, wherein the timestamp is included in the image frame to be transmitted to the host system.

13. The sensor unit of claim 1, wherein the original format of the image data is compatible with one of a YUV format or an RGB format.

14. The sensor unit of claim 1, wherein the sensor interface comprises an Ethernet interface to be coupled with a LIDAR device or one or more cameras.

15. The sensor unit of claim 1, wherein the sensor interface comprises a global positioning system (GPS) interface to be coupled to at least one of a GPS receiver and an IMU device.

16. The sensor unit of claim 1, wherein the sensor interface comprises a control area network (CAN) interface to be coupled to throttle control logic, braking control logic, and steering control logic of the ADV.

17. An autonomous driving system, comprising:

a plurality of sensors mounted on a plurality of locations of an autonomous driving vehicle (ADV);

a host system to perceive a driving environment surrounding the ADV based on sensor data obtained from the sensors and to plan a path to autonomously drive the ADV; and

a sensor unit coupled to the plurality of sensors and the host system, wherein the sensor unit comprises;

a sensor interface to be coupled to the plurality of sensors mounted on the ADV,

a host interface to be coupled to the host system, and

a plurality of data acquisition processors, each corresponding to one of the plurality of sensors, wherein each of the data acquisition processors includes;

a pixel alignment processor to reformat pixels of image data from an original format associated with a corresponding sensor to a predetermined format, and

a frame processing processor coupled to the pixel alignment processor, wherein the frame processing processor is configured to generate an image frame based on the image data received from the pixel alignment processor and to transmit the image frame to the host system.

18. The system of claim 17, wherein each data acquisition processor further comprises a sensor simulator processor coupled to the pixel alignment processor, wherein the sensor simulator processor is configured to simulate a sensor of the plurality of sensors to generate simulated image data without using the corresponding sensor.

19. The system of claim 18, wherein the simulated image data is utilized for testing a functionality of the pixel alignment processor and the frame processing processor without having to couple with a sensor.

20. The system of claim 17, wherein each data acquisition processor further comprises a host simulator processor coupled to the frame processing processor, wherein the host simulator processor is configured to simulate the host system by generating simulated host commands and responses without using the host system.