Video telephone systems

• US 5,543,939 A
• Filed: 09/02/1994
• Issued: 08/06/1996
• Est. Priority Date: 12/28/1989
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Harvey, Robert L., Hirschler-Marchand, Patrick R., Cipolle, David J.
• Primary Examiner(s): Coles, Sr., Edward L.
• Assistant Examiner(s): LEE, TOMMY D
• Application Number:
  US08/300,587
• Time in Patent Office: 704 Days
• Field of Search: 358/428, 358/429, 358/430, 358/261.2, 358/426, 358/455, 358/456, 358/457, 358/465, 358/466, 358/479, 348/407, 348/415, 348/390
• US Class Current: 375/240.02
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/4382 Demodulation or channel decoding, e.g. QPSK demodulation analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/440263 by altering the spatial resolution, e.g. for displaying on a connected PDA
• CPC Class Codes: H04N 7/12 Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal H04N7/24 takes precedence

Abstract

Differential image data compression systems and techniques are disclosed for use in video telephone systems for full duplex, real time transmission of audio and video images over nominal 3 kHz residential telephone lines and other narrow bandwidth channels. Each video telephone device consists of a transmitting portion and a receiving portion. In the transmitting section a reduced grey-scale (luminance) image, preferably consisting of only black-and-white pixels, is compared in an image processing module with a similarly reduced image derived from previous values to determine pixel positions that have changed. Information representative of the changes is then encoded and modulated and (time or frequency) multiplexed with the audio and/or chrominance signals for transmission. At the receiver, the incoming signal is demodulated and demultiplexed to separate the audio and video portions, the image portion is decoded and the luminance value is updated by an image updating unit. Adaptive resolution, pixel averaging and interpolation techniques are also disclosed for picture enhancement.

Claims

1. In a signal processing apparatus for video image data compression, a system comprising:

dithered threshold means for applying dithered threshold values to a multiple-level grey-scale image to convert said multiple grey-scale image into a reduced-level grey-scale image, the dithered threshold values varying spatially across an image and temporally across plural images;

storage means for storing a reference reduced grey-scale image derived from image pixel values;

comparison means for comparing a reduced grey-scale image of a current image frame with a reference image from said storage means, and for generating a luminance difference signal representative of the changes in reduced grey-scale pixel values between image frames;

encoding means for encoding said difference signal for transmission through a transmission system;

decoding means; and

reconstruction means for generating an expanded grey-scale image from the luminance difference signal having expanded levels from the reduced grey-scale image pixel values with substantially the same number of pixel values as in the reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image.

2. The system of claim 1 in which the encoding means further comprises a run-length encoding means for representing series of repeating differential image data bits in a coded fashion, such that long series of said bits are represented by fewer bits.

3. The system of claim 1 wherein the system further includes an adaptive resolution means for determining a level of motion in successive image frames and for modifying spatial resolution in response to such determination.

4. In a signal processing apparatus for video image data compression, a system comprising:

dithered threshold means for applying dithered threshold values to a multiple-level grey-scale image to convert said multiple grey-scale image into a reduced-level grey-scale image;

storage means for storing a reference reduced grey-scale image derived from image pixel values;

comparison means for comparing a reduced grey-scale image of a current image frame with a reference image from said storage means, and for generating a luminance difference signal representative of the changes in reduced grey-scale pixel values between image frames;

encoding means for encoding said difference signal for transmission through a transmission system;

decoding means;

reconstruction means for generating an expanded grey-scale image from the luminance difference signal having expanded levels from the reduced grey-scale image pixel values with substantially the same number of pixel values as in the reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image; and

wherein the dithered threshold values comprise sets of values varying spatially across an image, different sets representing different spatial resolutions, the system further comprising a resolution selector to select a set of dithered threshold values for the threshold means.

5. The system of claim 4 in which the dithered threshold means further comprises a hysteresis adjustment in which a threshold value applied to each pixel is modified in order to reduce toggling of the pixel value.

6. The system of claim 5 in which the system further comprises means for varying the hysteresis adjustment when a data rate of the encoded difference signal exceeds a limited bandwidth of a transmission medium.

7. In a signal processing apparatus for decoding and reconstructing a video image from received compressed image data, a system comprising:

a differential image decoding unit for decoding a difference signal representative of changes in a reduced grey-scale image from one frame to the next;

an image updating unit for updating changes to a previously stored image by adding said decoded difference signal to said previously stored image and thereby obtaining an updated image of reduced grey-scale pixel values; and

an image reconstruction unit for expanding each reduced grey-scale pixel value for each reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image, to increase the number of grey levels of said updated image with substantially the same number of pixel values as in the reduced grey-scale image, wherein the blocks of pixel values include pixel values from plural updated images for both spatial and temporal averaging.

8. The system of claim 7 which further comprises an image interpolating unit for generating a more detailed image by interpolation.

9. A method of signal processing for image data compression, the method comprising:

reducing the number of grey levels available to represent each pixel of an image frame by comparing a dithered threshold value to the luminance value of each pixel, and assigning the pixel a reduced grey-scale value based upon the comparison;

storing a reference reduced grey-scale image derived from image pixel values;

comparing a reduced grey-scale image of a current image frame with a previously stored reference reduced grey-scale image;

generating a luminance difference signal representative of changes in the reduced grey-scale pixel values between image frames;

reconstructing an expanded-level grey-scale image from the luminance difference signal with substantially the same number of pixel values in the reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image; and

storing sets of dithered threshold values, different sets representing different spatial resolutions, and selecting a set of dithered threshold values for the step of comparing.

10. The method of claim 9 in which the step of reducing the number of grey levels comprises converting a multiple grey-scale image into a halftone image.

11. The method of claim 9 in which the step of comparing a dithered threshold value further comprises applying a hysteresis adjustment to said threshold value in order to reduce toggling of the pixel value.

12. The method of claim 11 in which the method further comprises varying the hysteresis adjustment in response to the data rate of the encoded difference signal.

13. The method of claim 9 in which the step of encoding the luminance difference signal further comprises run-length encoding said signal such that commonly repeated series of image data bits are assigned shorter code words.

14. A method of signal processing for image data compression, the method comprising:

reducing the number of grey levels available to represent each pixel of an image frame by comparing a dithered threshold value to the luminance value of each pixel, and assigning the pixel a reduced grey-scale value based upon the comparison;

storing a reference reduced grey-scale image derived from image pixel values;

comparing a reduced grey-scale image of a current image frame with a previously stored reference reduced grey-scale image;

generating a luminance difference signal representative of changes in the reduced grey-scale pixel values between image frames;

reconstructing an expanded-level grey-scale image from the luminance difference signal with substantially the same number of pixel values in the reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image; and

storing dithered threshold values varying spatially across an image and temporally across plural images, and performing the step of comparing based on the spatially and temporally varying dithered threshold values.

15. The method of claim 14 further comprising storing sets of dithered threshold values, different sets representing different spatial resolutions, and selecting a set of dithered threshold values for the step of comparing.

16. A dithered threshold system for generating reduced grey-scale video images comprising:

sets of stored three dimensional arrays of dithered threshold values, each varying spatially across an image and varying temporally across plural images as an integral block of related dithered threshold values, different sets representing different resolutions;

threshold means for comparing grey-scale image pixel values of a sequence of video images to the stored dithered threshold values to generate reduced grey-scale image pixel values; and

a resolution selector for selecting a set of stored dithered threshold values for the threshold means.

17. A dithered threshold system for generating reduced grey-scale video images comprising:

a stored three dimensional array of dithered threshold values varying spatially across an image and varying temporally across plural images as an integral block of related dithered threshold values;

threshold means for comparing grey-scale image pixel values of a sequence of video images to the stored dithered threshold values to generate reduced grey-scale image pixel values; and

a reconstruction means for generating an expanded-level grey-scale image by combining reduced grey-scale image pixel values across a spatially and temporally extending block of reduced grey-scale pixel values.

18. The system of claim 17 wherein the stored three dimensional array of dithered threshold values further comprise sets of dithered threshold values, different sets representing different resolutions, and further comprising a resolution selector for selecting a set of stored dithered threshold values for the threshold means.

19. The system of claim 18 wherein the resolution selector comprises a motion detector for detecting motion between successive images.

20. The system of claim 19 wherein the motion detector detects motion based on a count of pixels which vary between images.

21. The system of claim 17 wherein the reconstruction means averages pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value.

22. A dithered threshold system for generating a reduced-level grey-scale image comprising:

stored sets of dithered threshold values varying spatially across an image, different sets representing different spatial resolutions;

a resolution selector comprising a motion detector for detecting motion between successive images; and

threshold means for comparing grey-scale image pixel values to a set of stored dithered threshold values selected by the resolution selector to generate reduced grey-scale image pixel values.

23. The system of claim 22 wherein the stored sets of dithered threshold values vary spatially across an image and temporally across plural images.

24. The system of claim 22 wherein the motion detector detects motion based on a count of pixels which vary between images.

25. A method of generating reduced video grey-scale images comprising:

storing sets of three dimensional arrays of dithered threshold values, each as an integral block of related dithered threshold values which vary spatially across an image and temporally across plural images, different sets representing different resolutions; and

comparing grey-scale image pixel values of a sequence of video images to the stored dithered threshold values to generate reduced grey-scale image pixel values; and

reconstructing an expanded grey-scale image by combining reduced grey-scale image pixel values across a spatially and temporally extending block of reduced grey-scale pixel values.

26. The method of generating reduced grey-scale video images comprising:

storing sets of dithered threshold values varying spatially across an image, different sets representing different spatial resolution;

selecting a set of dithered threshold values based on changes between successive video image frames which indicate movement between frames; and

comparing grey-scale image pixel values to a selected set of stored dithered threshold values to generate reduced grey-scale image pixel values.

27. In a signal processing apparatus for image data compression, a system comprising:

stored sets of dithered threshold values varying spatially across an image and temporally across plural images, different sets representing different spatial resolutions;

a resolution selector for selecting resolution based on content of successive grey-scale images;

dithered threshold means for applying dithered threshold values to a multiple-level grey-scale image to convert the multiple-level grey-scale image into a reduced-level grey-scale image;

storage means for storing a reference reduced grey-scale image derived from image pixel values;

comparison means for comparing a reduced grey-scale image of a current image frame with a reference image from the storage means, and for generating a luminous difference signal representative of changes in the reduced grey-scale pixel values between image frames;

run length encoding means for encoding the difference signal for transmission through a limited bandwidth transmission system;

decoding means; and

reconstruction means for generating an increased grey-scale image from the luminous difference signal, the reconstruction means including an averaging unit for expanding the grey-scale pixel values for each reduced grey-scale image by averaging pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value, to increase the number of grey levels of a reconstructed image.

28. A signal processing apparatus for expanding an image from reduced grey-scale image data comprising:

storage of pixel values of a reduced grey-scale image; and

an image expander for expanding each reduced grey-scale pixel value by averaging pixel values of respective blocks of pixel values, including pixel values from plural updated images for both spatial and temporal averaging, a different block of pixel values for each expanded pixel value defined by a shifting window over pixel values of the reduced grey-scale image, to increase the number of grey levels of each pixel with substantially the same number of pixel values as in the reduced grey-scale image.

29. In a signal processing apparatus for video image data compression, a system comprising:

dithered threshold means for applying dithered threshold values, which vary spatially across an image and temporally across plural images, to multiple-level grey-scale images to convert said multiple grey-scale images into reduced-level grey-scale images;

storage means for storing a reference reduced grey-scale image derived from image pixel values;

comparison means for comparing a reduced grey-scale image of a current image frame with a reference image from said storage means, and for generating a luminance difference signal representative of the changes in reduced grey-scale pixel values between image frames;

encoding means for encoding said difference signal for transmission through a transmission system;

decoding means; and

reconstruction means for generating an expanded grey-scale image from the luminance difference signal having expanded levels from the reduced grey-scale image pixel values.

30. In a signal processing apparatus for video image data compression, a system comprising:

dithered threshold means for applying dithered threshold values to a multiple-level grey-scale image to convert said multiple-level grey-scale image into a reduced-level grey-scale image, the dithered threshold values comprising sets of values varying spatially across an image, different sets representing different spatial resolutions;

a resolution selector to select a set of dithered threshold values for the threshold means;

storage means for storing a reference reduced grey-scale image derived for image pixel values;

comparison means for comparing a reduced grey-scale image of a current image frame with a reference image from said storage means, and for generating a luminance difference signal representative of the changes in reduced grey-scale pixel values between image frames;

encoding means for encoding said difference signal for transmission through a transmission system;

decoding means; and

reconstruction means for generating an expanded grey-scale image from the luminance difference signal having expanded levels from the reduced grey-scale image pixel values.

31. The system of claim 30 in which the dithered threshold means further comprises a hysteresis adjustment in which a threshold value applied to each pixel is modified in order to reduce toggling of the pixel value.

32. The system of claim 31 in which the system further comprises means for varying the hysteresis adjustment when a data rate of the encoded difference signal exceeds a limited bandwidth of a transmission medium.

33. In a signal processing apparatus for decoding and reconstructing a video image from received compressed image data, a system comprising:

a differential image decoding unit for decoding a difference signal representative of changes in a reduced grey-scale image from one frame to the next;

an image updating unit for updating changes to a previously stored image by adding said decoded difference signal to said previously stored image and thereby obtaining an updated image of reduced grey-scale pixel values; and

an image reconstruction unit for expanding each reduced grey-scale pixel value for each reduced grey-scale image by combining pixel values of respective blocks of pixel values, each block including pixel values from plural updated images for both spatial and temporal averaging and there being a different block of pixel values for each expanded pixel value, to increase the number of grey levels of said updated image.

34. A method of signal processing for image data compression, the method comprising:

storing sets of dithered threshold values, different sets representing different spatial resolutions, and selecting a set of dithered threshold values;

reducing the number of grey levels available to represent each pixel of an image frame by comparing a dithered threshold value of the selected set of dithered threshold values to the luminance value of each pixel, and assigning the pixel a reduced grey-scale value based upon the comparison;

storing a reference reduced grey-scale image derived from image pixel values;

comparing a reduced grey-scale image of a current image frame with a previously stored reference reduced grey-scale image;

generating a luminance difference signal representative of changes in the reduced grey-scale pixel values between image frames; and

reconstructing an expanded-level grey-scale image from the luminance difference signal.

35. A method of signal processing for image data compression, the method comprising:

storing dithered threshold values varying spatially across an image and temporally across plural images;

reducing the number of grey levels available to represent each pixel of an image frame by comparing a dithered threshold value of the spatially and temporally varying dithered threshold values to the luminance value of each pixel, and assigning the pixel a reduced grey-scale value based upon the comparison;

storing a reference reduced grey-scale image derived from image pixel values;

comparing a reduced grey-scale image of a current image frame with a previously stored reference reduced grey-scale image;

generating a luminance difference signal representative of changes in the reduced grey-scale pixel values between image frames; and

reconstructing an expanded-level grey-scale image from the luminance difference signal.

36. The method of claim 35 further comprising storing sets of dithered threshold values, different sets representing different spatial resolutions, and selecting a set of dithered threshold values for the step of comparing.

37. A dithered threshold system for generating a reduced grey-scale image comprising:

stored dithered threshold values varying spatially across an image and temporally across plural images;

threshold means for comparing grey-scale image pixel values to the stored dithered threshold values to generate reduced grey-scale image pixel values; and

reconstruction means for generating an expanded-level grey-scale image by combining reduced grey-scale image pixel values across a spatially and temporally extending block of reduced grey-scale pixel values.

38. The system of claim 37 wherein the stored dithered threshold values further comprise sets of dithered threshold values, different sets representing different resolutions, and further comprising a resolution selector for selecting a set of stored dithered threshold values for the threshold means.

39. The system of claim 38 wherein the resolution selector comprises a motion detector for detecting motion between successive images.

40. The system of claim 39 wherein the motion detector detects motion based on a count of pixels which vary between images.

41. The system of claim 37 wherein the reconstruction means averages pixel values of respective blocks of pixel values, a different block of pixel values for each expanded pixel value.

42. A method of generating a reduced grey-scale image comprising:

storing sets of dithered threshold values representing different spatial resolutions which vary spatially across an image and temporally across plural images;

comparing grey-scale image pixel values to the stored dithered threshold values to generate reduced grey-scale image pixel values; and

reconstructing an expanded grey-scale image by combining reduced grey-scale image pixel values across a spatially and temporally extending block of reduced grey-scale pixel values.

43. A signal processing apparatus for expanding an image from reduced grey-scale image data comprising:

storage of pixel values of a reduced grey-scale image; and

an image expander for expanding each reduced grey-scale pixel value by combining pixel values of respective blocks of pixel values, each block including pixel values from plural updated images for both spatial and temporal averaging and there being a different block of pixel values for each expanded pixel value, to increase the number of grey levels of each pixel.