System and method for increasing space or time resolution in video

• US 20050057687A1
• Filed: 06/17/2004
• Published: 03/17/2005
• Est. Priority Date: 12/26/2001
• Status: Active Grant

• Current Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Original Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Inventors: Irani, Michael, Shechtman, Eli, Caspi, Yaron
• Granted Patent: US 7,428,019 B2
• Time in Patent Office: Days
• US Class Current: 348/443
• CPC Class Codes: G06T 3/4069 by subpixel displacements
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/01 Conversion of standards , e.g. involving analogue television standards or digital television standards processed at pixel level
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/12 Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal H04N7/24 takes precedence

Abstract

A system and method for increasing space or time resolution of an image sequence by combination of a plurality input sources with different space-time resolutions such that the single output displays increased accuracy and clarity without the calculation of motion vectors. This system of enhancement may be used on any optical recording device, including but not limited to digital video, analog video, still pictures of any format and so forth. The present invention includes support for example for such features as single frame resolution increase, combination of a number of still pictures, the option to calibrate spatial or temporal enhancement or any combination thereof, increased video resolution by using high-resolution still cameras as enhancement additions and may optionally be implemented using a camera synchronization method.

Claims

1. A method for adjusting at least one aspect of the resolution, in time and/or space, of a space-time visual entity, comprising:

providing a plurality of space-time visual entities; and

combining information from said plurality of space-time visual entities to adjust at least one aspect of the resolution.

2. The method of claim 1, wherein each of said plurality of space-time visual entities comprises at least one of a sequence of images of a scene, a still image, a stream of data from visual detectors, video data of a scene or a combination thereof.

3. The method of claim 2, wherein each of said plurality of space-time visual entities comprises a sequence of images of a scene.

4. The method of claim 3, wherein the information is combined to form a single sequence of images.

5. The method of claim 2, wherein the resolution is increased in at least one of time and space by combining information from a plurality of lower resolution sources of sequences of images to construct a higher resolution sequence of images.

6. The method of claim 5, wherein said lower resolution sources of sequences of images feature different space-time resolutions.

7. The method of claim 6, wherein said lower resolution sources of sequences of images comprise video data from at least two of NTSC, PAL, HDTV, SECAM, a suitable video format or still images.

8. The method of claim 2, in which a plurality of sequence of images cameras is provided to obtain data of said scene.

9. The method of claim 8, wherein said providing said plurality of sources of sequence of images further comprises:

obtaining a relative physical location for said plurality of sequence of images cameras.

10. The method of claim 8, wherein said providing said plurality of sources of sequence of images sources further comprises:

obtaining temporal relations for said plurality of sequence of images.

11. The method of claim 1, further comprising:

determining a number of sequence of images cameras for increasing said temporal resolution.

12. The method of claim 9, wherein said relative physical location is obtained in a pre-calibration process.

13. The method of claim 9, wherein said obtaining further comprises detecting a misalignment between each pair of sequence of images cameras, and performing misalignment recovery for said misalignment.

14. The method of claim 8, wherein at least one sequence of images camera has a plurality of jointly built optics.

15. The method of claim 8, wherein at least two sequences of images cameras share joint optics.

16. The method of claim 8, wherein said providing said plurality of sequence of images sources further comprises:

obtaining a relative temporal order for said plurality of sequence of images.

17. The method of claim 13, wherein said misalignment at least comprises a temporal misalignment, such that said misalignment recovery comprises:

providing temporal synchronization hardware for synchronizing said plurality of sequence of images cameras.

18. The method of claim 17, wherein said temporal synchronization hardware performs a phase shift between sampling times of said plurality of sequence of images cameras.

19. The method of claim 13, wherein said misalignment at least comprises a temporal misalignment, such that said misalignment recovery comprises:

determining a correct temporal alignment from said sequence of images.

20. The method of claim 1, wherein the adjustments to the resolution are performed in both time and space.

21. The method of claim 1, wherein the resolution is increased to a greater degree in one of time or space.

22. The method of claim 1, wherein increasing resolution in time reduces one or more spatial artifacts.

23. The method of claim 1, wherein the adjustment to the resolution is performed at least by adjusting contrast.

24. The method of claim 23, wherein said contrast is adjusted by adjusting at least one of color resolution or gray-scale resolution.

25. The method of claim 24, wherein said at least one of said color resolution or said gray-scale resolution is enhanced.

26. The method of claim 23, wherein said contrast is adjusted by adjusting dynamic range.

27. The method of claim 26, wherein said contrast adjusting comprises increasing a number of bits per image element.

28. The method of claim 27, wherein said dynamic range is enhanced.

29. The method of claim 1, wherein the adjustment to the resolution comprises providing super-resolution.

30. The method of claim 29 wherein super-resolution comprises a capability to reconstruct high resolution frequencies, wherein said high resolution frequencies are not expressed in any of said plurality of space-time visual entities.

31. The method of claim 29, wherein super-resolution comprises spatial super-resolution.

32. The method of claim 29, wherein super-resolution comprises temporal super-resolution.

33. The method of claim 32 wherein said temporal super-resolution comprises resolving motion aliasing from said plurality of space-time visual entities.

34. The method of claim 32 wherein said temporal super-resolution comprises reducing motion blur from said plurality of space-time visual entities.

35. The method of claim 29 wherein said super-resolution is performed both in space and in time.

36. The method of claim 1, wherein each of said plurality of space-time visual entities comprises at least one unit, and wherein each unit comprises a plurality of sub-units, such that said plurality of space-time visual entities is misaligned at a sub-unit level, wherein said information is combined according to a sub-unit misalignment.

37. The method of claim 36, wherein said sub-unit misalignment comprises at least one of spatial or temporal misalignment.

38. The method of claim 37, wherein said sub-unit misalignment comprises both spatial and temporal misalignment.

39. The method of claim 36, wherein said sub-unit misalignment is predetermined.

40. The method of claim 1, wherein said plurality of space-time entities comprises a plurality of sources of sequence of images data, and each source of sequence of images of data has a first associated sampling rate in space and a second associated sampling rate in time, and wherein the resolution is increased according to a plurality of said first associated sampling rates in space and said second associated sampling rates in time.

41. The method of claim 40, wherein said combining said information from said plurality of sources of sequence of images data comprises:

determining a transformation according to at least one of space and time for transforming said plurality of sources of sequence of images data to higher resolution sequence of images data.

42. The method of claim 41, wherein said higher resolution sequence of images data is characterized by having a sampling rate in time and a sampling rate in space, and wherein said sampling rate in time and a sampling rate in space is higher than a highest of each of said plurality of said first associated sampling rates in space and said second associated sampling rates in time.

43. The method of claim 41, wherein said higher resolution sequence of images data is characterized by having a sampling rate in time and a sampling rate in space, and wherein said sampling rate in time in at least one of said sequence of images data is higher than a corresponding rate in each of a remaining sequence of images data, and wherein said sampling rate in time in at least one of said sequence of images data is higher than a corresponding rate in at least one sequence of images data other than sequence of images data having a high temporal sampling rate.

44. The method of claim 41, wherein said determining said transformation comprises:

determining at least one of spatial coordinates and temporal coordinates for each of said plurality of sources of sequences of images; and

determining an alignment between said at least one of spatial coordinates and temporal coordinates for each of said plurality of sources of sequences of images.

45. The method of claim 44, wherein said temporal coordinates are determined for each of said plurality of sources of sequences of images, and said transformation comprises a one-dimensional affine transformation in time.

46. The method of claim 44, wherein said spatial coordinates are determined for each of said plurality of sources of sequences of images, and said transformation comprises a two-dimensional projective transformation.

47. The method of claim 44, wherein said spatial coordinates are determined for each of said plurality of sources of sequences of images, and said transformation comprises:

recovering a relative physical location for obtaining each source of sequence of images.

48. The method of claim 46, wherein each source of sequence of images is obtained from a camera, and said transformation further comprises:

determining at least one calibration factor for each camera; and

adjusting said spatial coordinates according to said at least one calibration factor.

49. The method of claim 1, wherein a low resolution results from limitation of the imaging process and wherein resolution degradation in time and in space from said low resolution is modeled by convolution with a kernel.

50. The method of claim 49, wherein said kernel comprises a plurality of regional kernels for separate portions of said data from said plurality of sources of sequences of images.

51. The method of claim 50, wherein resolution degradation in space is modeled by a point spread function.

52. The method of claim 51, wherein said spatial point spread function varies along a field of view for said plurality of sources of sequences of images.

53. The method of claim 49, wherein the convolution kernel for each of said plurality of sources of sequences of images is known.

54. The method of claim 51, wherein a spatial point spread function for each of said plurality of sources sequences of images is approximated according to a pre-determined heuristic.

55. The method of claim 53, wherein a spatial point spread function for each of said plurality of sources sequences of images is measured in a pre-calibration step.

56. The method of claim 49, wherein a spatial point spread function for each of said plurality of sources of sequences of images is approximated according to a real-time calculation.

57. The method of claim 49, wherein resolution degradation in time is modeled by an exposure-time function as a temporal point spread function.

58. The method of claim 57, wherein a temporal spread function of each of said plurality of sources of sequences of images is known.

59. The method of claim 57, wherein said temporal spread function is determined by controlling an exposure time for each of said plurality of sources of sequences of images.

60. The method of claim 57, wherein said temporal spread function is approximated according to data from each of said plurality of sources of sequences of images.

61. The method of claim 1, wherein said visual space-time entities are input image sequences and the relation between the said input image sequences and the said high resolution output is expressed as an integral equation, and wherein the method further comprises solving said integral equation.

62. The method of claim 61, wherein said integral equation comprises:

63. The method of claim 61 where the said relation comprises both the said spatial and temporal transformations and said temporal and spatial spread functions.

64. The method of claim 1, wherein said visual space-time entities comprise a plurality of input image sequences and said solving said integral equation further comprises constructing a set of equations simulating equations of any of claims 61-63 and solving said equations, where the unknowns for said set of equations comprise an output sequence of images.

65. The method of claim 64, wherein said constructing is performed on data from each of said plurality of input image sequences by performing an approximation of a space-time point spread function for image elements from sources of input image sequences.

66. The method of claim 64, wherein said constructing is performed on data from each of said plurality of sources of sequences of images by transforming said data from each of said plurality of sources of sequences of images to a single coordinate system and performing one of uniform or non-uniform approximation on a space-time point spread function for each source for said transformation.

67. The method of claim 66, wherein an opposite of uniform or non-uniform approximations are performed for space, and for time.

68. The method of claim 64, wherein said combining said information from said plurality of sources of sequences of images further comprises:

solving a linear set of equations;

A{right arrow over (h)}={right arrow over (l)}

wherein {right arrow over (h)} is a vector containing color values for said sequences of images having an adjusted resolution, {right arrow over (l)} is a vector containing spatial and temporal measurements for said plurality of sources of sequences of images, and matrix A contains relative contributions of each space-time point of said sequences of images having said adjusted resolution to each space-time point from said plurality of sources of sequences of images.

69. The method of claim 61, wherein said equations further comprise a regularization term.

70. The method of claim 69, wherein said regularization term adds a preference to a smooth solution for said equations.

71. The method of claim 69, wherein said preference is expressed as smoothness along at least of the main axis.

72. The method of claim 69, wherein said preference is expressed as smoothness and is performed along space-time diagonal directions.

73. The method of claim 69, wherein said preference is expressed as having a low second derivative.

74. The method of claim 69, wherein said regularization term is weighted according to at least one weight.

75. The method of claim 74, wherein said at least one weight is determined according to data from said plurality of sources of sequences of images.

76. The method of claim 74, wherein said equations are solved iteratively, and said at least one weight is determined from a previous iteration.

77. The method of claim 74, wherein said at least one weight is determined according to first or second derivatives of data from said plurality of sources of sequences of images or a previous iteration, or a combination thereof.

78. The method of claim 69, wherein said space-time regularization term further comprises a directional regularization term according to

79. A method for treating a spatial artifact of sequences of images, comprising:

increasing the temporal resolution of the sequences of images.

80. The method of claim 79, wherein information is obtained from a plurality of cameras.

81. The method of claim 79, wherein each sequence comprises pixels, and the pixels of the output sequence have higher temporal resolution than the pixels of the input sequences.

82. A method for reducing motion blur of a plurality of sequences of images, comprising enhancing the temporal resolution of the plurality of sequences of images after capturing the plurality of sequences of images.

83. A method for reducing induced spatial blur in a sequence of images, comprising:

increasing the temporal resolution of the sequence of images.

84. A camera for providing an increased resolution sequence of images, comprising:

(a) a sequence of images camera for providing initial sequences of images;

(b) a still camera for providing still image data; and

(c) an integrator for integrating said still image data and said initial sequences of images for providing the sequence of images of increased resolution.

85. The camera of claim 84, wherein said still camera is located in physical proximity to said sequence of images camera.

86. The camera of claim 84, wherein said still camera and said sequence of images camera are located in the same device having the same optics.

87. The camera of claim 84, wherein said still and said sequence of images are generated by different sampling patterns of the same image sensor array.

88. The camera of claim 84, wherein said integrator comprises a computational device for receiving signals from each of said plurality of sequence of images and still cameras and for combining at least one aspect of the resolution, in time and space, of said signals, by combining information from said signals to adjust at least one aspect of the resolution.

89. The camera of claim 84, further comprising:

(d) a housing for containing said sequence of images camera and said still camera.

90. The camera of claim 84, wherein said integrator comprises a joint optical integrator.

91. The camera of claim 84, wherein said integrator comprises a chip.

92. The camera of claim 84, wherein said sensors have differing resolutions in at least one of time or space.

93. The camera of claim 84, wherein said sensors have at least one dissimilar property, said dissimilar property including at least one of spatial sampling rate, spatial point spread function, frame-rate, spatial resolution, dynamic range and exposure time.

94. The camera of claim 84, wherein the number of said plurality of sensors is determined according to the desired increase in resolution.

95. The camera of claim 84, further comprising a shutter, wherein said shutter opens and closes with a shutter speed such that temporal resolution of the sequences of images is blurred differently.

96. A camera system for providing increased resolution of sequences of images, comprising:

(a) a plurality of cameras, said plurality of cameras being physically separated; and

(b) a computational device for receiving signals from each of said plurality of cameras and for adjusting at least one aspect of the resolution, in time and/or space, of said signals, by combining information from said signals to adjust at least one aspect of the resolution.

97. The system of claim 96, wherein said plurality of cameras comprises a plurality of sequence of images cameras having similar properties.

98. The system of claim 97, wherein said similar properties include at least one of spatial resolution, frame-rate and exposure time.

99. The system of claim 96, wherein said plurality of cameras comprises a plurality of sequence of images cameras having at least one dissimilar property.

100. The system of claim 99, wherein said at least one dissimilar property includes at least one of spatial sampling rate, spatial point spread function, frame-rate, spatial resolution, dynamic range and exposure time.

101. The system of claim 96, wherein said plurality of sequence of images cameras comprises a plurality of video cameras having differing video formats.

102. The system of claim 96, wherein said plurality of cameras comprises at least one sequence of images camera and at least one still camera.

103. The system of claim 96, wherein said sensors have differing resolutions in at least one of time or space.

104. The system of claim 96, wherein said sensors have at least one dissimilar property, said dissimilar property including at least one of spatial sampling rate, spatial point spread function, frame-rate, spatial resolution, dynamic range and exposure time.

105. The system of claim 96, wherein the number of said plurality of sensors is determined according to the desired increase in resolution.

106. The system of claim 96, further comprising a shutter, wherein said shutter opens and closes with a shutter speed such that temporal resolution of the sequences of images is blurred differently.

107. A camera system for providing increased resolution of sequences of images, comprising:

(a) a plurality of sensors for capturing visual signals;

(b) an integrator for integrating said visual signals; and

(c) a computational device for receiving said integrated signals for increasing at least one aspect of the resolution, in time and/or space, of said signals, by combining information from said signals to increase at least one aspect of the resolution.

108. The system of claim 107, wherein said integrator comprises a joint optical set.

109. The system of claim 108, wherein said joint optical set comprises a beam-splitter.

110. The system of claim 107, wherein said sensors have differing properties, and wherein said integrator comprises a chip.

111. The system of claim 107, wherein said sensors produce data being shifted in at least one of time or space.

112. The system of claim 107, wherein said sensors have differing resolutions in at least one of time or space.

113. The system of claim 107, wherein said sensors have at least one dissimilar property, said dissimilar property including at least one of spatial sampling rate, spatial point spread function, frame-rate, spatial resolution, dynamic range and exposure time.

114. The system of claim 107, wherein said plurality of sensors is constructed as a single sensor and a plurality of detectors, wherein said plurality of detectors is shifted in at least one of time or space.

115. The system of claim 107, wherein the number of said plurality of sensors is determined according to the desired increase in resolution.

116. The system of claim 107, further comprising a shutter, wherein said shutter opens and closes with a shutter speed such that temporal resolution of the sequences of images is blurred differently.