Imaging surface modeling for camera modeling and virtual view synthesis

• US 9,858,639 B2
• Filed: 11/18/2015
• Issued: 01/02/2018
• Est. Priority Date: 10/11/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Wang, Jinsong, Litkouhi, Bakhtiar B., Lybecker, Kent S., Frakes, Ryan M.
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): SLATER, ALISON T
• Application Number:
  US14/944,409
• Publication Number:
  US 20160098815A1
• Time in Patent Office: 776 Days
• Field of Search: 348148, 382154
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2300/10 characterised by the type of camera system used
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30264 Parking
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method of displaying a captured image on a display device. A real image is captured by an image capture device. The image capture device uses a field-of-view lens that distorts the real image. A camera model is applied to the captured real image. The camera model maps objects in the captured real image to an image sensor plane of the image capture device to generate a virtual image. The image sensor plane is reconfigurable to virtually alter a shape of the image sensor plane to a non-planar surface. The virtual image formed on the non-planar image surface of the image sensor is projected to the display device.

Claims

1. A method of displaying a captured image on a display device comprising the steps of:

capturing a real image by an image capture device, the image capture device using a field-of-view lens that distorts the real image;

applying a camera model to the captured real image, the camera model mapping objects in the captured real image to an image sensor plane of the image capture device to generate a virtual image, the image sensor plane reconfigured to virtually alter a shape of the image sensor plane to a non-planar surface; and

projecting the virtual image formed on the non-planar image surface of the image sensor to the display device;

wherein generating the virtual image comprises the steps of;

providing a pre-calibrated real camera model by the processor, the real camera model representative of the vision-based imaging device capturing the scene;

determining real incident ray angles of each pixel in the captured image based on the pre-calibrated real camera model;

identifying an arbitrary shape of the non-planar imaging surface

identifying a pose of the virtual camera model;

determining virtual incident ray angles of each pixel in the virtual image based on the virtual image model and the non-planar imaging surface;

mapping a virtual incident ray to a correlated real incident ray of the real image capture device, wherein rotational compensation is applied to the virtual incident ray angles for correlating the virtual incident ray and the real incident ray if a pose of the virtual camera model is different from a pose of the of the real image capture device;

mapping pixels in the virtual image corresponding to coordinates on the non-planar virtual imaging surface to correlated pixels on the real captured image as a function of the correlation mapping between the real incident ray and the virtual incident ray, and wherein a horizontal projection of a virtual incident angle θ

_virt is represented by the angle α

, wherein angle α

is represented by the following formula;

2. The method of claim 1 wherein applying a camera model to the captured real image includes applying the camera model without radial distortion correction.

3. The method of claim 1 wherein a respective real image pixel on the real image is represented by coordinates u_ureal and v_real for identifying where the real incident ray is imaged on a real image surface, wherein the real incident ray is represented by the angles (θ

_real, φ

_real), wherein θ

_real is an angle between the real incident ray and an optical axis of a real image capture device, and φ

_real is the angle between a real image capture device x axis and a projection of the real incident ray on a real image capture device x-y plane.

4. The method of claim 3 wherein the real incident ray angle is determined based on a predefined and calibrated real camera model.

5. The method of claim 4 wherein the real camera model includes a fisheye camera model, wherein the fisheye camera model represents a fisheye lens having substantial radial distortion for generating an ultra-wide angle field-of-view scene on a flat imaging plane with limited sensor size, wherein the correlation between the real incident ray and a real image pixel on the real image sensor from the real image capture device is represented as follows:

6. The method of claim 5 wherein a radial distortion correction is applied to the fisheye camera model, wherein a radial distortion is determined by a distortion effect of a lens of the vision-based imaging device, wherein the radial distortion is represented by a function of the distorted radial distance with respect to an undistorted variable, wherein a distorted radial distance is measured from a respective image pixel to an image center on an image sensor of the real image capture device using the lens with distortion effect, wherein the un-distorted variable includes an incident ray angle of the incident chief ray which is imaged to the distorted image pixel, wherein the radial distortion for vision-based imaging device having a field-of-view greater than 135 degrees is represented by the following formula:

r_d=p₁·

θ

₀+p₂·

θ

₀³+p₃·

θ

₀⁵+ . . .

where p are distortion parameters.

7. The method of claim 4 wherein a radial distortion correction is applied to a real camera model when radial distortion is present, wherein the radial distortion is represented by a function of the distorted radial distance with respect to an undistorted variable, wherein the undistorted variable includes an undistorted radial distance of an image pixel based on a pin-hole camera model, and wherein the radial distortion for vision-based imaging device having a field-of-view less than 135 degrees is represented by the following formula:

r_d=r₀(1+k₁·

r₀²+k₂·

r₀⁴+k₂·

r₀⁶+ . . . )

where point r₀ is the undistorted radial distance, wherein r₀ is determined using a pinhole model and includes intrinsic and extrinsic parameters, and wherein k are radial distortion parameters.

8. The method of claim 4 wherein a real camera is modeled as a pinhole model with a non-planar imaging surface, wherein the pinhole model represents a lens without radial distortion, and wherein a radial distortion effect of a real camera lens is modeled into the non-planar imaging surface.

9. The method of claim 1 wherein a vertical projection of a virtual incident angle incident angle θ

_virt is represented by the angle β

, wherein angle β

is represented by the following formula;

10. The method of claim 9 wherein the virtual incident ray angles are determined by the following formulas:

11. The method of claim 1 wherein if the pose of the virtual camera model is different than the real camera model, then θ

_virt is an angle between the virtual incident ray and the virtual optical axis represented by a virtual z axis, and φ

_virt is an angle between a virtual camera x axis and a projection of the virtual incident ray on the virtual camera model x-y plane, wherein any point on the virtual incident ray is represented by the following matrix;

12. The method of claim 11 wherein the difference between the pose of the virtual camera model and the pose of the real camera model is represented by a rotation matrix.

13. The method of claim 12 wherein the coordinates of a same point on the correlated real incident ray and the virtual incident ray for the identified pose difference between the virtual camera model and the real camera model is represented by the following matrix:

14. The method of claim 13 wherein real incident ray angles (θ

_real,φ

_real) is determined by the following formula;

15. The method of claim 1 wherein a virtual incident ray angles (θ

_virt,φ

_virt) is equal to a real incident ray (θ

_real,φ

_real) if the pose of the virtual camera model and the real camera model are the same.

16. The method of claim 1 wherein the virtual camera model includes a pin-hole camera model.

17. The method of claim 1 wherein the non-planar imaging surface includes an elliptical imaging surface.

18. The method of claim 1 wherein the non-planar imaging surface includes a cylindrical imaging surface.

19. The method of claim 1 wherein the non-planar imaging surface includes a spherical imaging surface.

20. The method of claim 1 wherein the non-planar imaging surface is an arbitrary shape that satisfies a homographic mapping from respective image pixels projected on the non-planar image surface to respective incident rays.

21. The method of claim 20 wherein the non-planar imaging surface is dynamically switched to another arbitrary shape based on a driving scenario for a respective vehicle operation, wherein the selected arbitrary shape enhances the virtual image projected on the display device for the respective vehicle operation.

22. The method of claim 1 wherein the vision-based imaging device is a rear back-up camera.

23. The method of claim 1 wherein the vision-based imaging device is a side-view camera.

24. The method of claim 1 wherein the vision-based imaging device is a front-view camera.

25. The method of claim 1 wherein generating the virtual camera model includes defining a virtual camera pose, wherein a difference between in the pose of a virtual camera model and a pose of the vision-based imaging device is modeled as a rotation matrix.

26. The method of claim 25 wherein the difference in the pose of the virtual camera pose and the pose of the vision-based imaging device is modeled by rotating the image surface, wherein rotating the image surface maintains the same pose between the virtual camera model and the vision-based imaging device pose.

27. The method of claim 1 wherein generating the virtual imaging comprises the steps of:

constructing a coordinate system for a respective camera model, that includes a x-axis, a-axis, and a z-axis, the z-axis being aligned with a camera optical axis extending outward the respective camera model, the x-axis and the y-axis forming a plane perpendicular to the z-axis, the x-axis and the z-axis intersecting at a camera aperture location;

defining the camera pose as the location of the camera coordinates and the orientation of the camera z-axis;

generating a real camera model that is representative of the vision-based imaging device, the real image being a scene captured by the vision-based imagining device;

generating a virtual camera model including simulated camera model parameters, a simulated imaging surface, and a simulated camera pose;

identifying a virtual image that is a synthesized image of a scene using the virtual camera model;

selecting a shape of the non-planar imaging surface;

selecting a pose of the virtual camera model;

determining a virtual incident ray angle for each pixel in the virtual image based on the virtual camera model using the non-planar imaging surface;

determining whether the pose of the virtual camera model is the same as a pose of the real camera model in response to comparing the virtual incident ray angle and the real incident real angle;

applying a rotational matrix for correlating the virtual incident angle and the real incident angle in response to a difference in the poses of the virtual camera model and the virtual camera model, the rotational matrix representing a transformation for correlating the virtual camera pose to the real camera pose;

determining a corresponding pixel location by applying the real camera model with angular distortion parameters, wherein the real camera model with angular distortion represents an imaging process of the vision-based imaging device capturing a scene, and wherein parameters of the real camera model with angular distortion are determined by a camera calibration process;

generating a mapping between a respective virtual image pixel location and a respective real image pixel location as a function of a respective virtual image pixel coordinate, the virtual incident ray angle, the real incident ray angle, and the real image pixel coordinate; and

synthesizing the virtual image of the scene from the real image, wherein a pixel value of the real image is utilized as the pixel value for a corresponding pixel in the virtual image.

28. The method of claim 27 wherein the synthesizing the virtual image of the scene from the real image includes interpolating of the pixel values of neighboring pixels of the corresponding location in the real image for determining the corresponding pixel value in the virtual image.

29. The method of claim 1 wherein a dynamic view synthesis for generating the virtual image is enabled based on a driving scenario of a vehicle operation, wherein the dynamic view synthesis generates a direction zoom to a region of the image for enhancing visual awareness to a driver for the respective region.

30. The method of claim 29 wherein the driving scenario of a vehicle operation for enabling the dynamic view synthesis includes determining whether the vehicle is driving in a parking lot.

31. The method of claim 30 wherein the driving scenario of a vehicle operation for enabling the dynamic view synthesis includes determining whether the vehicle is driving in on highway.

32. The method of claim 31 wherein the driving scenario of a vehicle operation for enabling the dynamic view synthesis includes actuating a turn signal.

33. The method of claim 32 wherein the driving scenario of a vehicle operation for enabling the dynamic view synthesis is based on a steering wheel angle.

34. The method of claim 33 wherein the driving scenario of a vehicle operation for enabling the dynamic view synthesis is based on a speed of the vehicle.