Digital multi-view video compression with complexity and compatibility constraints

• US 6,055,012 A
• Filed: 12/27/1996
• Issued: 04/25/2000
• Est. Priority Date: 12/29/1995
• Status: Expired due to Term

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Haskell, Barin Geoffry, Puri, Atul, Kollarits, Richard Virgil
• Primary Examiner(s): LEE, RICHARD J
• Application Number:
  US08/777,334
• Time in Patent Office: 1,215 Days
• Field of Search: 348/42, 348/45-60, 348/564, 348/43, 348/402, 348/407, 348/409-413, 348/415, 348/416, 348/699, 351/201
• US Class Current: 348/48
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

In a system and method for transmitting and displaying multiple different views of a scene, three or more simultaneous scene signals, representing multiple different views of a scene, are provided by an appropriate camera arrangement to a spatial multiplexer. The size and resolution of the scene signals are reduced as necessary to combine the multiple scene signals into two super-view signals. The super-view signals are encoded using compression based on redundancies between the two super-views and then transmitted. A decoder receives the transmitted data signal and extracts the two super-view signals according to the inverse of the encoding operation. A spatial demultiplexer recovers the individual scene signals from the decoded super-view signals in accordance with a defined multiplexed order and arrangement. The scene signals are then interpolated as needed to restore the original resolution and size and subsequently displayed.

Claims

1. A system, for coding and decoding multiple simultaneous scene signals representing multiple different views of a scene obtained from a camera arrangement, comprising:

a spatial view multiplexer which combines more than two scene signals into super-view 1 and super-view 2 signals and generates an arrangement data signal indicating a particular multiplex arrangement and order of said more than two scene signals in said super-view 1 and super-view 2 signals;

an encoder and system multiplexer assembly which receives said super-view 1 and super-view 2 signals and said arrangement data signal, encodes said super-view 1 and super-view 2 signals using compression based on redundancies between the two super-views to produce encoded super-view 1 and encoded super-view 2 signals, and combines said encoded super-view 1 signal, said encoded super-view 2 signal, and said arrangement data signal into a multiplexed data stream;

a transmission channel for transmitting said multiplexed data stream;

a decoder and system demultiplexer assembly which receives said multiplexed data stream, separates said encoded super-view 1 signal, said encoded super-view 2 signal, and said arrangement data signal from the multiplexed data stream, and operates on said encoded super-view 1 and super-view 2 signals to decode them according an inverse operation of the encoding in said encoder to produce decoded super-view 1 and decoded super-view 2 signals; and

a spatial view demultiplexer which receives said decoded super-view 1 an decoded super-view 2 signals and said arrangement data signal and separates scene signals from the decoded super-view 1 and decoded super-view 2 signals in accordance with said particular multiplex arrangement and order indicated by said arrangement data signal to provide multiple output scene signals representing said multiple different views of the scene.

2. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 1 wherein said encoder and said decoder are disparity compensated.

3. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 1 wherein said encoder and said decoder are motion compensated and disparity compensated.

4. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 3 wherein said super-view 1 signal is encoded and decoded independently with motion compensation and said super-view 2 signal is encoded and decoded with motion compensation and disparity compensation based on said super-view 1 signal.

5. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 1 wherein said encoder and decoder operate according to MPEG-2 with disparity compensation.

6. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 5 wherein said super-view 1 signal is encoded and decoded independently according to an MPEG-2 main profile and said super-view 2 signal is encoded and decoded according to an MPEG-2 temporal scalability process with disparity compensation based on said super-view 1 signal.

7. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 1 wherein there are at least four simultaneous scene signals in the form of video picture frames each having a particular horizontal and vertical resolution, and further comprising:

a decimator which reduces one of the horizontal and vertical resolution of at least one scene signal by at least a factor of two to produce reduced resolution video picture frame signals,

said spatial view multiplexer combining the reduced resolution video picture frame signals into adjacent areas of at least one of said super-view 1 and super-view 2 signals.

8. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 7 wherein said decimator horizontally decimates the scene signal and simultaneously filters the scene represented by said scene signal by using a digital filter.

9. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 7 wherein said decimator vertically decimates the scene signal by a factor of two by dropping alternate field lines to achieve a progressive frame of half height.

10. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 7 wherein said decimator vertically decimates the scene signal and simultaneously filters the scene represented by said scene signal by using a digital filter to achieve a progressive frame of half height.

11. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 7 wherein said decimator horizontally and vertically decimates the scene signal by a factor of two to achieve a progressive frame of half height and half width.

12. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 2 further including a spatial interpolator after the spatial view demultiplexer, said interpolator restoring the resolution to the decoded scene signals which was reduced by the decimator.

13. A system for coding and decoding multiple simultaneous scene signals as claimed in claim 12 wherein said interpolator horizontally and/or vertically restores the resolution by a factor of two using a digital interpolation filter.

14. A system for generating and displaying multiple simultaneous scene signals representing multiple different views of a scene, comprising:

a camera arrangement directed at a scene and producing more than two simultaneous scene signals representing multiple different views of the scene;

a spatial view multiplexer which combines said more than two simultaneous scene signals into super-view 1 and super-view 2 signals according to a particular multiplex arrangement and order and generates an arrangement data signal indicating said particular multiplex arrangement and order;

an encoder which receives said super-view 1 and super-view 2 signals and said arrangement data signal and applies an encoding operation to said super-view 1 and super-view 2 signals using compression based on redundancies between the two super-views to produce encoded super-view 1 and encoded super-view 2 signals, said encoder further including said arrangement data signal within at least one of said encoded super-view 1 and encoded super-view 2 signals;

a system multiplexer which combines said encoded super-view 1 and encoded super-view 2 signals into a multiplexed data stream, at least one of said encoded super-view 1 and super-view 2 signals including said arrangement data signal;

a transmission channel for transmitting said multiplexed data stream;

a system demultiplexer which receives said multiplexed data stream and separates said encoded super-view 1 and encoded super-view 2 signals from the multiplexed data stream;

a decoder which receives said encoded super-view 1 and encoded super-view 2 signals from said system demultiplexer, extracts said arrangement data signal from said at least one of said encoded super-view 1 and encoded super-view 2 signals, and applies a decoding operation to said encoded super-view 1 and encoded super-view 2 signals according to an inverse operation of the encoding operation in said encoder to produce decoded super-view 1 and decoded super-view 2 signals;

a spatial view demultiplexer which receives said decoded super-view 1 and decoded super-view 2 signals and said arrangement data signal and separates scene signals from the decoded super-view 1 and decoded super-view 2 signals in accordance with said particular multiplex arrangement and order indicated by said arrangement data signal to provide multiple output scene signals representing said multiple different views of the scene; and

a display which simultaneously displays said multiple output scene signals.

15. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the camera arrangement is a single camera with multiple lenses to capture multiple views.

16. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the camera arrangement is a plurality of cameras arranged in parallel with each other at different locations and directed to the scene.

17. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the camera arrangement comprises multiple stereo camera arrangements, each said stereo camera including at least a left and right camera aligned with each other and directed at the scene at a particular angle, the particular angle of a left camera being equal to and opposite the angle of the corresponding right camera with respect to a centerline between them directed at the scene.

18. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the camera arrangement is a plurality of cameras arranged in a convex configuration within a scene and directed at different angles towards the scene.

19. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the camera arrangement is a plurality of cameras arranged in a concave configuration outside a scene and directed at different angles towards the scene.

20. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the display comprises:

a view selector which sequentially selects scene signals;

a projector which converts the scene signals into light signals; and

a screen which receives the light signals and sequentially displays the scenes.

21. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 17 wherein the display comprises:

a left view selector which selects scene signals from the left camera of a stereo camera arrangement;

a right view selector which selects scene signals from the corresponding right camera of a stereo camera arrangement;

a left projector which converts the scene signals from the left view selector into light signals;

a right projector which converts the scene signals from the right view selector into light signals;

a screen which receives the light signals from the left and right projectors and displays the scenes;

a left polarizing lens located between the left projector and said screen and which polarizes the light signals from said left projector;

a right polarizing lens located between the right projector and said screen and which polarizes the light signals from said right projector; and

glasses with a left polarizing lens over one eye of a user and a right polarizing lens over the other eye of the user.

22. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 18 wherein the display comprises:

a concave screen with a number of viewing areas related to the number of cameras; and

a number of projectors related to the number of cameras and spaced from corresponding viewing areas along an arc related to the concave structure of the screen, viewers of the display being located between the projectors and the screen areas.

23. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 19 wherein the display comprises:

a convex screen with a number of viewing areas related to the number of cameras; and

a number of projectors related to the number of cameras and spaced from corresponding viewing areas along an arc related to the convex structure of the screen, viewers of the display being located between the projectors and the screen areas.

24. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 20

wherein the screen is an auto-stereoscopic lenticular display;

and

the projector includes the view selector and projects stripes of alternate views based on the scene signals.

25. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein the display comprises:

a view selector which sequentially selects scene signals; and

a cathode ray tube monitor which converts the scene signals into a light display of the scenes.

26. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein said encoder and said decoder are disparity compensated.

27. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein said encoder and said decoder are motion compensated and disparity compensated.

28. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 27 wherein said super-view 1 signal is encoded and decoded independently with motion compensation and said super-view 2 signal is encoded and decoded with motion compensation and disparity compensation based on said super-view 1 signal.

29. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein said encoder and decoder operate according to MPEG-2 with disparity compensation.

30. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 29 wherein said super-view 1 signal is encoded and decoded independently according to an MPEG-2 main profile and said super-view 2 signal is encoded and decoded according to an MPEG-2 temporal scalability process with disparity compensation based on said super-view 1 signal.

31. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 14 wherein there are at least four simultaneous scene signals in the form of video picture frames each having a particular horizontal and vertical resolution, and further comprising:

a decimator which reduces one of the horizontal and vertical resolution of at least one scene signal by at least a factor of two to produce reduced resolution video picture frame signals, said spatial view multiplexer combining the reduced resolution video picture frame signals into adjacent areas of at least one of said super-view 1 and super-view 2 signals.

32. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 31 wherein said decimator horizontally decimates the scene signal and simultaneously filters the scene represented by said signal using a digital filter.

33. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 31 wherein said decimator vertically decimates the scene signal by a factor of two by dropping alternate field lines to achieve a progressive frame of half height.

34. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 31 wherein said decimator vertically decimates the scene signal and simultaneously filters the scene represented by said scene signal using a digital filter.

35. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 31 wherein said decimator horizontally and vertically decimates the scene signal by a factor of two to achieve a progressive frame of half height and half width.

36. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 26 further including a spatial interpolator after the demultiplexer, said interpolator restoring the resolution to the decoded scene signals which was reduced by the decimator.

37. A system for generating and displaying multiple simultaneous scene signals as claimed in claim 36 wherein said interpolator horizontally and/or vertically restores the resolution by a factor of two using a digital interpolation filter.

38. A method of coding and decoding multiple simultaneous scene signals representing multiple different views of a scene obtained from a camera arrangement, comprising the steps of:

spatially multiplexing more than two scene signals into super-view 1 and super-view 2 signals in a particular multiplexed arrangement and order;

generating an arrangement data signal indicating said particular multiplexed arrangement and order used during said spatially multiplexing step;

encoding said super-view 1 and super-view 2 signals using compression based on redundancies between the two super-views to produce encoded super-view 1 and encoded super-view 2 signals;

transmitting on a transmission channel said encoded super-view 1 and encoded super-view 2 signals and said arrangement data signal;

receiving said encoded super-view 1 and encoded super-view 2 signals and said arrangement data signal from said transmission channel;

decoding said encoded super-view 1 and encoded super-view 2 signals according an inverse operation of the encoding to produce decoded super-view 1 and decoded super-view 2 signals; and

spatially demultiplexing scene signals from the decoded super-view 1 and decoded super-view 2 signals in accordance with said particular multiplex arrangement and order as indicated by said arrangement data signal to provide multiple output scene signals representing said multiple different views of the scene.

39. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 38 wherein there are at least four simultaneous scene signals in the form of streams of video picture frames, the picture frames in each stream each having a particular horizontal and vertical resolution and a corresponding picture size defined by the number of pixels in the horizontal and vertical directions, the method further comprising the steps of:

reducing one of the horizontal and vertical resolutions and the corresponding number of pixels in the horizontal or vertical direction of the picture frames in at least one scene signal by at least a factor of two to produce reduced resolution video picture frame signals prior to the spatially multiplexing step, and

combining the reduced resolution video picture frame signals into adjacent areas of at least one of said super-view 1 and super-view 2 signals.

40. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 39 wherein the step of reducing involves reducing the horizontal resolution of the video picture frames in the scene signal by a factor of two using a decimating filter to achieve one of an interlaced or progressive frame of half width.

41. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 39 wherein the step of reducing involves reducing the vertical resolution of the video picture frames in the scene signal by a factor of two by dropping alternate field lines to achieve a progressive frame of half height.

42. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 39 wherein the step of reducing involves reducing the vertical resolution of the video picture frames in the scene signal by a factor of two by using a decimating filter to achieve a progressive frame of half height.

43. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 39 wherein the step of reducing involves reducing the horizontal resolution and vertical resolution of the video picture frames in the scene signal by a factor of two to achieve a progressive frame of half height and half width.

44. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 39 further including the step of spatial interpolation after the step of spatially demultiplexing, said interpolating causing restoration of the horizontal or vertical picture size to the picture frames in the decoded scene signals which was reduced by the step of reducing the horizontal or vertical resolution and further causing at least partial restoration of the resolution by correcting at least some of the artifacts introduced by the steps of reducing and interpolation.

45. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 44 wherein said step of interpolating uses a digital interpolating filter and increases the number of pixels in the picture frames in the decoded scene signals in one of the horizontal and vertical direction thereby increasing the picture size by a factor of two.

46. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 38 wherein the steps of encoding and decoding are performed with disparity compensation.

47. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 38 wherein the steps of encoding and decoding are performed with motion compensation and disparity compensation.

48. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 47 wherein said super-view 1 signal is encoded and decoded independently with motion compensation and said super-view 2 signal is encoded and decoded with motion compensation and disparity compensation based on said super-view 1 signal.

49. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 38 wherein the steps of encoding and decoding are performed according to MPEG-2 with disparity compensation.

50. A method of coding and decoding multiple simultaneous scene signals as claimed in claim 49 wherein said super-view 1 signal is encoded and decoded independently according to an MPEG-2 main profile and said super-view 2 signal is encoded and decoded according to an MPEG-2 temporal scalability process with disparity compensation based on said super-view 1 signal.

51. A method of generating and displaying multiple simultaneous scene signals representing multiple different views of a scenes comprising the steps of:

directing a camera arrangement at a scene and producing more than two simultaneous scene signals representing multiple different views of the scene, each scene signal comprising a stream of video picture frames;

spatially multiplexing said more than two simultaneous scene signals into super-view 1 and super-view 2 signals in a particular multiplexed arrangement and order;

generating an arrangement data signal indicating said particular multiplexed arrangement and order used during said spatially multiplexing step;

encoding said super-view 1 and super-view 2 signals using compression based on redundancies between the two super-views to produce encoded super-view 1 and encoded super-view 2 signals;

transmitting said encoded super-view 1 and encoded super-view 2 signals and said arrangement data signal;

receiving said encoded super-view 1 and encoded super-view 2 signals and said arrangement data signal;

decoding said encoded super-view 1 and encoded super-view 2 signals according to an inverse of the encoding in order to produce decoded super-view 1 and decoded super-view 2 signals;

spatially demultiplexing scene signals from the decoded super-view 1 and decoded super-view 2 signals in accordance with said particular multiplexed arrangement and order as indicated by said arrangement data signal to provide multiple output scene signals representing said multiple different views of the scene; and

displaying simultaneously said multiple output scene signals.

52. A method of generating and displaying multiple simultaneous scene signals as claimed in claim 51 wherein the camera arrangement is a single camera with multiple lenses to capture multiple views.