System and method for fisheye image processing

• US 10,796,402 B2
• Filed: 10/19/2018
• Issued: 10/06/2020
• Est. Priority Date: 10/19/2018
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Yan, Zhipeng, Chen, Pengfei, Wang, Panqu
• Primary Examiner(s): Bekele, Mekonen T
• Application Number:
  US16/165,951
• Publication Number:
  US 20200126179A1
• Time in Patent Office: 718 Days
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/249 from positioning sensors located off-board the vehicle, e.g. from cameras
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

A system and method for fisheye image processing is disclosed. A particular embodiment can be configured to: receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame; partition the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame; warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view; combine the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame; generate auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

Claims

1. A system comprising:

a data processor; and

a fisheye image processing system, executable by the data processor, the fisheye image processing system being configured to;

receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;

partition the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame;

warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;

combine the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame;

generate auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and

provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

2. The system of claim 1 being further configured to perform object extraction on the combined resulting fisheye image data set to identify extracted objects in the fisheye image frame.

3. The system of claim 1 being further configured to align the mapped target view with a line parallel to a side of the autonomous vehicle.

4. The system of claim 1 wherein the at least one fisheye lens camera is from the group consisting of:

a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

5. The system of claim 1 being further configured to obtain parameters of the fisheye lens, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

6. The system of claim 1 being further configured to map pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set.

7. The system of claim 1 being further configured to map pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set and to combine all of the pixel mappings to produce an end-to-end mapping for optimization.

8. A method comprising:

receiving fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;

partitioning the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame;

warping each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;

combining the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame;

generating auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and

providing the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

9. The method of claim 8 including performing object extraction on the combined resulting fisheye image data set to identify extracted objects in the fisheye image frame.

10. The method of claim 8 including aligning the mapped target view with a line parallel to a side of the autonomous vehicle.

11. The method of claim 8 wherein the at least one fisheye lens camera is from the group consisting of:

a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

12. The method of claim 8 including obtaining parameters of the fisheye lens, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

13. The method of claim 8 including mapping pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set.

14. The method of claim 8 including mapping pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set and combining all of the pixel mappings to produce an end-to-end mapping for optimization.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;

partition the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame;

warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;

combine the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame;

generate auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and

provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

16. The non-transitory machine-useable storage medium of claim 15 being further configured to perform object extraction on the combined resulting fisheye image data set to identify extracted objects in the fisheye image frame.

17. The non-transitory machine-useable storage medium of claim 15 being further configured to align the mapped target view with a line parallel to a side of the autonomous vehicle.

18. The non-transitory machine-useable storage medium of claim 15 wherein the at least one fisheye lens camera is from the group consisting of:

a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

19. The non-transitory machine-useable storage medium of claim 15 being further configured to obtain parameters of the fisheye lens, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

20. The non-transitory machine-useable storage medium of claim 15 being further configured to map pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set.