Method apparatus and computer program products for performing perspective corrections to a distorted image

• US 6,346,967 B1
• Filed: 10/28/1999
• Issued: 02/12/2002
• Est. Priority Date: 05/27/1994
• Status: Expired due to Term

• Current Assignee: BH Image Co LLC (Intellectual Ventures LLC)
• Original Assignee: Be Here Corporation
• Inventors: Gullichsen, Eric, Wyshynski, Susan
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US09/429,697
• Time in Patent Office: 838 Days
• Field of Search: H04/N.5225, 348/207, 348/218, 348/222, 348/335, 348/340, 348/143, 348/500, 348/552, 382/300, 382/128, 382/293, 382/44
• US Class Current: 348/207.99
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A method and apparatus for transforming wide angle video into perspective corrected viewing zones which either a single user or multiple users may select, orient and magnify. The present invention first captures a wide angle digital video input by any suitable means. The captured image is then stored in a suitable memory means so portions of the image may be selected at a later time. When a portion of the stored video is selected for viewing, a plurality of discrete viewing vectors in three dimensional space are chosen on the video input and transformed to a plurality of control points in a two dimensional plane or any other suitable surface. The area between these points which is still warped from the original wide angle image capture is then transformed to a perspective corrected field of view. The perspective corrected field of view is then displayed on a suitable displaying apparatus, such as a monitor or head mounted display.

Claims

1. An apparatus having a central processing unit (CPU) and a memory coupled to said CPU to perform perspective correction comprising:

an image capture mechanism configured to capture a distorted image through a wide-angle lens;

a selection mechanism configured to select a set of control vectors into a set of control points using a function that models said wide-angle lens;

a function generator configured to generate a polynomial transform function that maps said set of control points into a set of reference points; and

a transform engine configured to transform a sub area of said distorted image proximate to said set of control points using said polynomial transform function.

2. The apparatus of claim 1 wherein said polynomial transform function comprises a global bivariate polynomial transformation.

3. The apparatus of claim 1 wherein said polynomial transform function comprises a biquadratic polynomial transformation.

4. The apparatus of claim 1 wherein said set of control vectors comprise a principal viewing and a plurality of surrounding vectors that form a planar view.

5. The apparatus of claim 4 wherein said planar view is a rectangular view.

6. The apparatus of claim 1 wherein said wide-angle lens is a fish eye lens.

7. The apparatus of claim 1 wherein said set of reference points is a set of rectangular points.

8. The apparatus of claim 1 further comprising a filter stage configured to apply one or more filters to said sub area responsive to said transform engine.

9. A computer program product including:

a computer usable data carrier having computer readable code embodied therein for causing a computer to perform perspective correction, said computer readable code including;

computer readable program code configured to cause said computer to effect an image capture mechanism for capturing a distorted image through a wide-angle lens;

computer readable program code configured to cause said computer to effect a selection mechanism configured to select a set of control vectors into a set of control points using a function that models said wide-angle lens;

computer readable program code configured to cause said computer to effect a function generator configured to generate a polynomial transform function that maps said set of control points into a set of reference points; and

computer readable program code configured to cause said computer to effect a transform engine configured to transform a sub area of said distorted image proximate to said set of control points using said polynomial transform function.

10. The computer program product of claim 9 wherein the computer usable data carrier is a computer readable media.

11. The computer program product of claim 9 wherein the computer usable data carrier is a carrier wave.

12. The computer program product of claim 9 wherein said polynomial transform function comprises a global bivariate polynomial transformation.

13. The computer program product of claim 9 wherein said polynomial transform function comprises a biquadratic polynomial transformation.

14. The computer program product of claim 9 wherein said set of control vectors comprise a principal viewing and a plurality of surrounding vectors that form a planar view.

15. The computer program product of claim 14 wherein said planar view is a rectangular view.

16. The computer program product of claim 9 wherein said wide-angle lens is a fish eye lens.

17. The computer program product of claim 9 wherein said set of reference points is a set of rectangular points.

18. The computer program product of claim 9 further comprising computer readable program code configured to cause said computer to effect a filter stage configured to apply one or more filters to said sub area responsive to said transform engine.

19. An apparatus for transforming wide angle video signals from live, prerecorded or computer simulated environments into perspective corrected video signals, the apparatus comprising:

an image capture mechanism configured to capture a wide angle video input;

a memory electrically coupled to the image capture mechanism to store said wide angle video input;

a processor electrically coupled to the memory and configured to transform a plurality of viewing vectors to a plurality of control points on a portion of said wide angle video input, the processor further configured to transform an area between said plurality of control points to a perspective corrected field of view;

a presentation mechanism electrically coupled to the memory and configured to present said perspective corrected field of view; and

a selection mechanism electrically coupled to the processor and configured to select one or more portions of said wide angle video input for the processor to transform.

20. The apparatus of claim 19 wherein the presentation mechanism further includes a television, a video monitor, a computer display, a video recorder, or a computer storage media.

21. An apparatus for correcting the perspective of wide angle video, including:

an image capture mechanism configured to receive video information comprising a wide angle view of a scene;

a selection mechanism configured to allow a viewer to identify a plurality of viewing vectors as an area of interest within said wide angle view captured by the image capture mechanism;

a digitizer configured to digitize at least a portion of said wide angle view;

a selection mechanism configured to associate a plurality of control points with said area of interest identified by the viewer;

a transformation mechanism configured to transform an area associated with said plurality of control points into a perspective corrected field of view; and

a presentation mechanism configured to present said perspective corrected field of view to the viewer.

22. The apparatus of claim 21 wherein the presentation mechanism further includes a television, a video monitor, a computer display, a video recorder, or a computer storage media.

23. An apparatus for correcting the perspective of wide angle video data comprising:

an image capture mechanism configured to receive and store video information representing a wide angle view;

a selection mechanism configured to identify a plurality of viewing vectors as a first area of interest among said video information;

an association mechanism responsive to the selection mechanism configured to associate with the area of interest a plurality of control points; and

a mapping mechanism responsive to the selection mechanism configured to transform a second area associated with the control points from the wide angle view into a predetermined second field of view.

24. The apparatus of claim 23 wherein the first area and second area are substantially congruent.

25. A method for providing perspective corrected images of live, prerecorded or simulated environments from a sequence of image frames, the method comprising steps of:

receiving one of a plurality of distorted images from said sequence of image frames;

storing a portion of said one of said plurality of distorted images;

transforming a set of control vectors to a set of control points that defines an area that associates said portion of said one of said plurality of distorted images with a portion of a perspective corrected image;

transforming said portion of said one of said plurality of distorted images associated with said area to said portion of said perspective corrected image;

displaying said portion of said perspective corrected image;

sensing inputted information; and

controlling the transformation and display of said perspective corrected image through said inputted information.

26. The method of claim 25 wherein said set of control points contains a number of control points, said number having a lower limit of five and an upper limit of one-half of the number of pixels in said portion of said perspective corrected image.

27. The method of claim 25 wherein said number is nine.

28. The method of claim 25 wherein said sequence of image frames is a video stream.

29. The method of claim 25 wherein said sequence of image frames is received from a video playback device.

30. The method of claim 25 wherein said sequence of image frames is received from a computer storage device.

31. The method of claim 25 wherein said sequence of image frames is received from a video camera.

32. The method of claim 31 wherein said video camera is a digital video camera.

33. The method of claim 31 wherein said video camera is an analog video camera coupled to a video digitizer.

34. The method of claim 25 wherein said sequence of image frames is received from a network.

35. The method of claim 34 wherein said network is a computer network.

36. The method of claim 34 wherein said network is in communication with the Internet.

37. The method of claim 34 wherein said network is a broadcast network.

38. The method of claim 25 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one wide-angle lens.

39. The method of claim 25 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one fisheye lens.

40. The method of claim 25 wherein said one of said plurality of distorted images includes an image of at least one sixth of an environment.

41. An apparatus for transforming a portion of a distorted image within a sequence of image frames from live, prerecorded or computer simulated environments into a portion of a perspective corrected image, the apparatus comprising:

an input configured to receive one of a plurality of distorted images from said sequence of image frames;

a memory, coupled to the input, configured to store a portion of said one of said plurality of distorted images;

a processor, coupled to the memory, configured to transform a set of control vectors to a set of control points that defines an area that associates said portion of said one of said plurality of distorted images with a portion of a perspective corrected image, the processor further configured to transform said portion of said one of said plurality of distorted images associated with said area to said portion of said perspective corrected image;

a presentation mechanism, coupled to the memory, configured to present said portion of said perspective corrected image; and

a selection mechanism, coupled to the processor, configured to specify said set of control vectors.

42. The apparatus of claim 41 wherein said set of control points contains a number of control points, said number having a lower limit of five and an upper limit of one-half of the number of pixels in said portion of said perspective corrected image.

43. The apparatus of claim 41 wherein said number is nine.

44. The apparatus of claim 41 wherein said sequence of image frames is a video stream.

45. The apparatus of claim 41 wherein said sequence of image frames is received from a video playback device.

46. The apparatus of claim 41 wherein said sequence of image frames is received from a computer storage device.

47. The apparatus of claim 41 wherein said sequence of image frames is received from a video camera.

48. The apparatus of claim 47 wherein said video camera is a digital video camera.

49. The apparatus of claim 47 wherein said video camera is an analog video camera coupled to a video digitizer.

50. The apparatus of claim 41 wherein said sequence of image frames is received from a network.

51. The apparatus of claim 50 wherein said network is a computer network.

52. The apparatus of claim 50 wherein said network is in communication with the Internet.

53. The apparatus of claim 50 wherein said network is a broadcast network.

54. The apparatus of claim 41 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one wide-angle lens.

55. The apparatus of claim 41 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one fisheye lens.

56. The apparatus of claim 41 wherein said one of said plurality of distorted images includes an image of at least one sixth of an environment.

57. A computer program product including:

a computer usable data carrier having computer readable code embodied therein for causing a computer to transform a portion of a distorted image within a sequence of image frames from live, prerecorded or computer simulated environments into a portion of a perspective corrected image, said computer readable code including;

computer readable program code configured to cause said computer to effect an input mechanism configured to receive one of a plurality of distorted images from said sequence of image frames;

computer readable program code configured to cause said computer to effect a storage mechanism configured to store a portion of said one of said plurality of distorted images received by the input mechanism;

computer readable program code configured to cause said computer to effect a transformation mechanism configured to transform a set of control vectors to a set of control points that defines an area that associates said portion of said one of said plurality of distorted images with a portion of a perspective corrected image, the processor further configured to transform said portion of said one of said plurality of distorted images associated with said area to said portion of said perspective corrected image;

computer readable program code configured to cause said computer to effect a presentation mechanism, configured to present said portion of said perspective corrected image generated by the transformation mechanism; and

computer readable program code configured to cause said computer to effect a selection mechanism configured to specify said set of control vectors.

58. The computer program product of claim 57 wherein the computer usable data carrier is a computer readable media.

59. The computer program product of claim 57 wherein the computer usable data carrier is a carrier wave.

60. The computer program product of claim 57 wherein said set of control points contains a number of control points, said number having a lower limit of five and an upper limit of one-half of the number of pixels in said portion of said perspective corrected image.

61. The computer program product of claim 57 wherein said sequence of image frames is a video stream.

62. The computer program product of claim 57 wherein said sequence of image frames is received from a video playback device.

63. The computer program product of claim 57 wherein said sequence of image frames is received from a computer storage device.

64. The computer program product of claim 57 wherein said sequence of image frames is received from a video camera.

65. The computer program product of claim 57 wherein said sequence of image frames is received from a network.

66. The computer program product of claim 65 wherein said network is a computer network.

67. The computer program product of claim 65 wherein said network is in communication with the Internet.

68. The computer program product of claim 65 wherein said network is a broadcast network.

69. The computer program product of claim 57 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one wide-angle lens.

70. The computer program product of claim 57 wherein said one of said plurality of distorted images is a result of imaging an environment through at least one fisheye lens.

71. The computer program product of claim 57 wherein said one of said plurality of distorted images includes an image of at least one sixth of an environment.