Low bit rate audio-visual communication system having integrated preceptual speech and video coding

• US 5,548,322 A
• Filed: 05/31/1995
• Issued: 08/20/1996
• Est. Priority Date: 04/06/1994
• Status: Expired due to Term

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Zhou, Yong
• Primary Examiner(s): Kostak, Victor R.
• Application Number:
  US08/455,809
• Time in Patent Office: 447 Days
• Field of Search: 348/14, 348/17, 348/20, 348/484, 348/152, 348/462
• US Class Current: 348/14.01
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G10L 19/012 Comfort noise or silence coding
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/62 by frequency transforming in three dimensions H04N19/63 takes precedence
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

Disclosed is a low bit rate audio and video communication system which employs an integrated encoding system that dynamically allocates available bits among the audio and video signals to be encoded based on the content of the audio and video information and the manner in which the audio and video information will be perceived by a viewer. A dynamic bit allocation and encoding process will evaluate the current content of the audio and video information and allocate the available bits among the audio and video signals to be encoded. In addition, an appropriate audio encoding technique is dynamically selected based on the current content of the audio signal. A face location detection subroutine will detect and model the location of faces in each video frame, in order that the facial regions may be more accurately encoded than other portions of the video frame. A lip motion detection subroutine will detect the location and movement of the lips of a person present in a video scene, in order to determine when a person is speaking and to encode the lip regions more accurately. The audio and video signals generated by a second party to a communication are monitored to determine if the second party is paying attention to the audio and video information transmitted by the first party to the communication.

Claims

1. A method of encoding a first audio signal and a first video signal in a transmitter of an audio-visual communication system associated with a first party for transmission utilizing a number of available bits to a receiver associated with a second party, said receiver decoding said first audio and video signals for presentation to said second party, said encoding method comprising the steps of:

detecting whether said second party is watching a display screen; and

allocating a predetermined minimal number of said number of available bits for encoding said first video signal if said detecting step determines that said second party is not watching said display screen.

2. The encoding method according to claim 1, wherein said detecting step comprises the steps of:

performing a face location detection subroutine on a second video signal associated with said second party at a reference time when said second party is known to be looking in the direction of said camera in order to detect and model said face of said second party, said face being modelled with an ellipse defined by parametric values including a reference major axis and a reference minor axis, said major axis divided by said minor axis resulting in a reference aspect ratio;

performing said face location detection subroutine on said second video signal at a test time to measure said major axis and said minor axis of said modelled face ellipse, said measured major axis divided by said measured minor axis resulting in a current measured aspect ratio;

performing a lip motion detection subroutine on said second video signal at said test time in order to detect and model said lips, said lips being modelled with an ellipse defined by parametric values including a major axis and a minor axis; and

concluding that said second party is not looking in said general direction of said camera at said test time if one of said following conditions occur;

said face location detection subroutine performed at said test time has determined that there is no face in said second video signal;

the difference between said current measured aspect ratio and said reference aspect ratio exceeds a predefined threshold;

or

the distance between said minor axis of said lip ellipse and said measured major axis of said face ellipse exceed a predefined threshold.

3. The method of claim 1 wherein the first video signal comprises a plurality of image frames, and said detecting step further comprises testing a current video frame to determine whether said second party is watching the display screen.

4. The method of claim 3 further comprising the step of:

setting a no person watching flag in a speaker data base if said testing step determines said second party is not watching the display screen.

5. The method of claim 3 further comprising the step of:

setting a person watching flag in a speaker data base if said testing step determines said second party is watching the display screen.

6. The method of claim 1 further comprising the step of:

initializing a speaker data base by setting a no speaker watching flag in the speaker data base for each speaker in a videoconference.

7. The method of claim 6, wherein the step of detecting whether said second party is watching the display screen further comprises the step of checking whether a no speaker watching flag is set in the speaker data base for each speaker in the videoconference.

8. The method of claim 1 further comprising the step of:

initializing a speaker data base by storing an aspect ratio for a face modelling ellipse which models the face of said second party.

9. The method of claim 8, wherein the first video signal comprises a plurality of image frames, and said detecting step further comprises measuring an aspect ratio of a face ellipse detected in a current frame;

and comparing said measured aspect ratio with the stored aspect ratio.

10. The method of claim 9 further comprising the steps of:

computing a difference between said measured and stored aspect ratios; and

determining if said difference exceeds a threshold value.

11. The method of claim 1, wherein the first video signal comprises a plurality of image frames, and said detecting step further comprises:

analyzing a number of said plurality of image frames; and

changing the allocation of said number of available bits for encoding upon detecting that said second party is watching said display screen.