System and method for converting a fish-eye image into a rectilinear image

• US 8,670,001 B2
• Filed: 11/30/2006
• Issued: 03/11/2014
• Est. Priority Date: 11/30/2006
• Status: Expired due to Fees

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Comer, Robert P., Sharma, Anurag
• Primary Examiner(s): Sams, Michelle L
• Application Number:
  US11/565,436
• Publication Number:
  US 20080129723A1
• Time in Patent Office: 2,658 Days
• Field of Search: 345/419, 345/653, 345/679, 345/619, 345/664, 345/647
• US Class Current: 345/619
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

A system and method are disclosed for converting a region of a Fish-Eye or other wide-angle image into a rectilinear image. According to an illustrative embodiment of the present disclosure, a Fish-Eye to rectilinear image conversion block is defined for use in a block diagram of a diagrammatic programming environment. The Fish-Eye to rectilinear image conversion block may use any of a variety of models for Fish-Eye systems, and thus is “model-independent.” Further, intuitive zoom, pan, tilt, and attitude adjustment parameters may be used with the image conversion block, to allow control of the conversion by users lacking advanced understanding of optics.

Claims

1. A method for converting a region of a wide-angle image into a rectilinear image, the method comprising:

converting, by a first processor coupled to a memory storing at least a portion of the region of a wide-angle image, a location of a pixel in the rectilinear image into a 3-D position relative to the rectilinear image;

in response to a pan parameter, applying, by the first or a second processor, a first rotation matrix to the 3-D position;

in response to a tilt parameter, applying, by the first, second or a third processor, a second rotation matrix to the 3-D position;

mapping the 3-D position to a location of one or more pixels in the wide-angle image; and

setting attributes of the pixel in the rectilinear image using attributes of the one or more pixels in the wide angle image.

2. The method of claim 1, wherein the wide-angle image is a Fish-Eye image.

3. The method of claim 2, in which mapping the 3-D position is performed using a model of a wide-angle system selected from the group consisting of an equidistant model, a stereographic model, an equi-solid angle model, and an orthogonal model.

4. The method of claim 1, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal, the first rotation matrix performing a rotation about a first axis orthogonal to an optical axis of the image acquisition device.

5. The method of claim 4, further comprising:

the second rotation matrix performing a rotation about a second axis that is orthogonal to the optical axis of the image acquisition device, and that is also orthogonal to the first axis.

6. The method of claim 1, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately vertical down-looking, the first rotation matrix performing a rotation about an optical axis of the image acquisition device.

7. The method of claim 6, further comprising:

the second rotation matrix performing a rotation about an axis orthogonal to the optical axis of the image acquisition device.

8. The method of claim 1, further comprising:

in response to a first adjustment parameter, applying a third rotation matrix to the 3-D position to rotate in a first direction prior to the application of the first and second rotation matrices; and

in response to a second adjustment parameter, applying a fourth rotation matrix to the 3-D position to rotate in a second direction prior to the application of the first and second rotation matrices.

9. The method of claim 8, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal, the third and fourth fort-rotation matrices aligning the 3-D position in accord with approximately true horizontal.

10. The method of claim 9, wherein the third rotation matrix performs a rotation about an optical axis of the image acquisition device.

11. The method of claim 10, wherein the fourth rotation matrix performs a rotation about an axis orthogonal to the optical axis of the image acquisition device.

12. The method of claim 8, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately down-looking vertical, the first and second rotation matrices aligning the 3-D position in accord with approximately true down-looking vertical.

13. The method of claim 12, wherein the third rotation matrix performs a rotation about a first axis that is orthogonal to an optical axis of the image acquisition device.

14. The method of claim 13, wherein the fourth rotation matrix performs a rotation about a second axis that is orthogonal to an optical axis of the image acquisition device, and that is also orthogonal to the first axis.

15. The method of claim 1, further comprising:

specifying, by a user, the pan and tilt parameters by manipulating an input device.

16. The method of claim 15, wherein the input device is a joystick.

17. The method of claim 1, wherein the method is implemented by a graphical image conversion component within a diagrammatic programming environment.

18. The method of claim 1, wherein the method is implemented by functionality of a text-based programming environment.

19. The method of claim 1, wherein the method is implemented by an embedded hardware target.

20. An apparatus for converting a region of a wide-angle image into a rectilinear image, the apparatus comprising:

an image acquisition device configured to obtain the wide-angle image;

a computer-implemented representation of a rectilinear system configured to convert a location of a pixel in the rectilinear image into a 3-D position relative to the rectilinear image;

a first rotation matrix configured to rotate the 3-D position in response to a pan parameter;

a second rotation matrix configured to rotate the 3-D position in response to a tilt parameter;

a computer-implemented representation of a wide-angle system configured to map the 3-D position to a location of one or more pixels in the wide-angle image and set attributes of the pixel in the rectilinear image using attributes of the one or more pixels in the wide angle image; and

a display device configured to display the rectilinear image.

21. The apparatus of claim 20, wherein the wide-angle image is a Fish-Eye image.

22. The apparatus of claim 21, wherein the computer-implemented representation of a wide-angle system includes a model of a wide-angle system selected from the group consisting of an equidistant model, a stereographic model, an equi-solid angle model, and an orthogonal model.

23. The apparatus of claim 20, wherein the first rotation matrix is further configured to, in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal, perform a rotation about a first axis orthogonal to an optical axis of the image acquisition device.

24. The apparatus of claim 23, wherein the second rotation matrix is further configured to perform a rotation about a second axis that is orthogonal to the optical axis of the image acquisition device, and that is also orthogonal to the first axis.

25. The apparatus of claim 20, wherein the first rotation matrix is further configured to, in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately vertical down-looking, perform a rotation about an optical axis of an image acquisition device.

26. The apparatus of claim 25, wherein the second rotation matrix is further configured to perform a rotation about an axis orthogonal to the optical axis of the image acquisition device.

27. The apparatus of claim 20, further comprising:

a third rotation matrix configured to rotate the 3-D position in a first direction; and

a fourth rotation matrix configured to rotate the 3-D position in a second direction.

28. The apparatus of claim 27, wherein the third and fourth rotation matrices are configured to align the 3-D position in accord with approximately true horizontal in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal.

29. The apparatus of claim 27, wherein the third and fourth rotation matrices are configured to align the 3-D position in accord with approximately true down-looking vertical, in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately down-looking vertical.

30. The apparatus of claim 20 further comprising:

an input device configured to specify the pan and tilt parameters in response to manipulation by a user.

31. The apparatus of claim 30, wherein the input device is a joystick.

32. The apparatus of claim 30, wherein the computer-implemented representation of a rectilinear system and the computer-implemented representation of a wide-angle system are implemented by a graphical image conversion component within a diagrammatic programming environment.

33. The apparatus of claim 30, wherein the computer-implemented representation of a rectilinear system and the computer-implemented representation of a wide-angle system are implemented by functionality of a text-based programming environment.

34. The apparatus of claim 30, wherein the computer-implemented representation of a rectilinear system and the computer-implemented representation of a wide-angle system are implemented by an embedded hardware target.

35. A non-transitory computer readable medium containing executable program instructions for converting a region of a wide-angle image into a rectilinear image, the executable program instructions comprising program instructions configured to:

convert a location of a pixel in the rectilinear image into a 3-D position relative to the rectilinear image;

in response to a pan parameter, apply a first rotation matrix to the 3-D position;

in response to a tilt parameter, apply a second rotation matrix to the 3-D position;

map the 3-D position to a location of one or more pixels in the wide-angle image; and

set attributes of the pixel in the rectilinear image using attributes of the one or more pixels in the wide angle image.

36. The non-transitory computer readable medium of claim 35, wherein the wide-angle image is a Fish-Eye image.

37. The non-transitory computer readable medium of claim 36, in which mapping the 3-D position is performed using a model of a wide-angle system selected from the group consisting of an equidistant model, a stereographic model, an equi-solid angle model, and an orthogonal model.

38. The non-transitory computer readable medium of claim 35, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal, the first rotation matrix performing a rotation about a first axis orthogonal to an optical axis of the image acquisition device.

39. The non-transitory computer readable medium of claim 38, further comprising:

the second rotation matrix performing a rotation about a second axis that is orthogonal to the optical axis of the image acquisition device, and that is also orthogonal to the first axis.

40. The non-transitory computer readable medium of claim 35, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately vertical down-looking, the first rotation matrix performing a rotation about an optical axis of the image acquisition device.

41. The non-transitory computer readable medium of claim 40, further comprising:

the second rotation matrix performing a rotation about an axis orthogonal to the optical axis of the image acquisition device.

42. The non-transitory computer readable medium of claim 35, further comprising:

in response to a first adjustment parameter, applying a third rotation matrix to the 3-D position to rotate in a first direction prior to the application of the first and second rotation matrices; and

in response to a second adjustment parameter, applying a fourth rotation matrix to the 3-D position to rotate in a second direction prior to the application of the first and second rotation matrices.

43. The non-transitory computer readable medium of claim 42, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately horizontal, the third and fourth rotation matrices aligning the 3-D position in accord with approximately true horizontal.

44. The non-transitory computer readable medium of claim 43, wherein the third rotation matrix performs a rotation about an optical axis of the image acquisition device.

45. The non-transitory computer readable medium of claim 44, wherein the fourth rotation matrix performs a rotation about an axis orthogonal to the optical axis of the image acquisition device.

46. The non-transitory computer readable medium of claim 42, further comprising:

in response to an indication that an image acquisition device that captured the wide-angle image was oriented approximately down-looking vertical, the first and second rotation matrices aligning the 3-D position in accord with approximately true down-looking vertical.

47. The non-transitory computer readable medium of claim 46, wherein the third rotation matrix performs a rotation about a first axis that is orthogonal to an optical axis of the image acquisition device.

48. The non-transitory computer readable medium of claim 47, wherein the fourth rotation matrix performs a rotation about a second axis that is orthogonal to an optical axis of the image acquisition device, and that is also orthogonal to the first axis.

49. The non-transitory computer readable medium of claim 35, further comprising:

specifying, by a user, the pan and tilt parameters by manipulating an input device.

50. The non-transitory computer readable medium of claim 49, wherein the input device is a joystick.

51. The non-transitory computer readable medium of claim 35, wherein the executable program instructions are implemented by a graphical image conversion component within a diagrammatic programming environment.

52. The non-transitory computer readable medium of claim 35, wherein the executable program instructions are implemented by functionality of a text-based programming environment.

53. The non-transitory computer readable medium of claim 35, wherein the executable program instructions are implemented by an embedded hardware target.

54. An apparatus for converting a wide-angle image into a rectilinear image, the apparatus comprising:

a wide angle camera mounted to a vehicle and configured to capture a wide angle image;

a display device internal to the vehicle configured to display a rectilinear image; and

a processor configured to map, using a mathematical representation of a rectilinear system and a mathematical representation of a wide angle system, a location of each pixel in the rectilinear image to a location of one or more pixels in the wide-angle image, set attributes of each pixel in the rectilinear image from the one or more pixels in the wide angle image, and convert the location of each pixel to a 3-D position relative to the rectilinear image.

55. The apparatus of claim 54 wherein the processor is further configured to apply a first rotation matrix to rotate the 3-D position in response to a pan parameter, apply a second rotation matrix to rotate the 3-D position in response to a tilt parameter, and map the 3-D position to the one or more pixels in the wide-angle image.

56. The apparatus of claim 55 further comprising:

an input device internal to the vehicle configured to specify the pan and tilt parameters in response to manipulation by a user.