Panel transform

• US 10,453,173 B2
• Filed: 07/24/2015
• Issued: 10/22/2019
• Est. Priority Date: 07/24/2015
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH, Robert Bosch LLC (Robert Bosch GmbH)
• Original Assignee: Robert Bosch GmbH
• Inventors: Baran, Matthew T.
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US14/808,144
• Publication Number:
  US 20170024851A1
• Time in Patent Office: 1,551 Days
• Field of Search: None
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

System and method of performing various transformations between an omnidirectional image model and a rectilinear image model for use in vehicle imaging systems. In particular, a panel transform system and method for transforming an omnidirectional image from an omnidirectional camera positioned on a vehicle to a rectilinear image. The rectilinear image is then displayed to the driver for use while performing vehicle maneuvers. The panel transform system and method also provide a rectilinear image model based on the omnidirectional camera for use with existing rectilinear imaging processing systems. The rectilinear image model is determined based on a variable set of input parameters that are defined by both automatic and manual system inputs such as a steering wheel angle sensor and a user interface.

Claims

1. A method of converting an omnidirectional image from an omnidirectional camera located on a vehicle to a rectilinear image by a control unit, the method comprising:

accessing an omnidirectional camera model representative of an omnidirectional image on a unit sphere in an optical coordinate system, the omnidirectional camera model having a first plurality of first pixels described in reference to an optical origin of the optical coordinate system mapped to a real-world coordinate system;

receiving a set of parameters defining a local plane tangent to the unit sphere, the local plane representative of a rectilinear camera model having a second plurality of second pixels;

determining a plane extent of the local plane based on the set of parameters, the plane extent defining a boundary of the local plane;

determining a panel model relationship between the first plurality of first pixels and the second plurality of second pixels, wherein the panel model is determined by determining a sample spacing of the second plurality of second pixels based on an angular sample spacing of the omnidirectional camera model and the set of parameters;

receiving, at the control unit, the omnidirectional image from the omnidirectional camera; and

generating a rectilinear image from the omnidirectional image based on the panel model relationship between the first plurality of first pixels of the unit sphere and the second plurality of second pixels of the local plane, wherein the panel model relationship maps a portion of the omnidirectional image to the rectilinear image by warping a location of each of the first pixels of the omnidirectional image to a corresponding second pixel location of the rectilinear image.

2. The method of claim 1, wherein receiving the set of parameters defining the local plane tangent to the unit sphere includes receiving information describing a vector originating at an origin of the optical coordinate system and pointing to a point of contact between the local plane and the unit sphere.

3. The method of claim 2, wherein the information describing the vector includes a plurality of angles.

4. The method of claim 3, wherein the vector defines a line of sight from the optical origin to a region of interest on the omnidirectional image.

5. The method of claim 2, wherein the set of parameters further includes a field of view and a resolution of the local plane.

6. The method of claim 1, wherein defining a boundary of the local plane further includes defining a complete perimeter of the local plane with respect to the optical coordinate system.

7. The method of claim 1, wherein the local plane tangent to the unit sphere defines a region of interest in the omnidirectional image.

8. The method of claim 7, further comprising mapping a plurality of locations on the unit sphere to a world location described by world coordinates.

9. The method of claim 8, wherein each coordinate point on the unit sphere grid defines each pixel location of the first plurality of first pixels, and wherein each corresponding coordinate point on the local plane grid defines each corresponding pixel location of the second plurality of second pixels.

10. The method of claim 9, wherein generating the rectilinear image includes copying information from each of the pixel locations of the omnidirectional image to locations on the local plane.

11. The method of claim 1, further comprising calibrating the mapping of the plurality of locations on the unit sphere with a predefined calibration pattern captured within the omnidirectional image.

12. The method of claim 1, wherein determining the panel model relationship between the first plurality of first pixels and the second plurality of second pixels includes

determining a local plane grid that defines coordinate points on the local plane with respect to the optical coordinate system;

and

determining a unit sphere grid that defines coordinate points on the unit sphere with respect to the optical coordinate system by projecting the coordinate points of the local plane grid onto the unit sphere.

13. The method of claim 1, further comprising displaying the rectilinear image on a display in the vehicle.

14. The method of claim 1, wherein receiving a set of parameters defining a local plane tangent to a unit sphere includes receiving the set of parameters from an automated vehicle system, wherein the set of parameters received from the automated vehicle system automatically defines an area of interest of the omnidirectional image.

15. The method of claim 1, wherein receiving a set of parameters defining a local plane tangent to a unit sphere includes determining the set of parameters by the control unit based on driver input.

16. The method of claim 1, wherein generating a rectilinear image is performed by projecting a portion of the omnidirectional image from the unit sphere onto the local plane based on the set of parameters.

17. The method of claim 1, further comprising

receiving a second set of parameters defining a second local plane tangent to the unit sphere, the second local plane representative of a second rectilinear camera model having a third plurality of third pixels;

determining a relationship between the first plurality of first pixels and the third plurality of third pixels;

receiving, at the control unit, the omnidirectional image from the omnidirectional camera; and

generating a second rectilinear image from the omnidirectional image based on the relationship between the first plurality of first pixels and the third plurality of third pixels, wherein the second rectilinear image includes pixels that are mapped to the real-world coordinate system

wherein the second rectilinear image has a different perspective view than the rectilinear image.

18. The method of claim 17, further comprising displaying the second rectilinear image on a display in the vehicle.

19. The method of claim 1, wherein determining the panel model relationship between the first plurality of first pixels and the second plurality of second pixels is performed, in part, through interpolation using look-up tables.