Video transcoder, video transcoding method, and video communication system and method using video transcoding with dynamic sub-window skipping

• US 6,711,212 B1
• Filed: 09/22/2000
• Issued: 03/23/2004
• Est. Priority Date: 09/22/2000
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Chia-Wen
• Primary Examiner(s): LEE, Y YOUNG
• Application Number:
  US09/667,538
• Time in Patent Office: 1,278 Days
• Field of Search: H04/N.712, 375/240.01, 375/240.12, 375/240.16, 375/240.24, 375/240.26, 375/240.27, 375/240.29, 382/232, 382/235, 382/238, 709/232, 348/14.01, 348/14-7-, 348/14.15
• US Class Current: 375/240.24
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A multipoint video conferencing system employs a transcoder with a dynamic sub-window skipping technique to enhance the visual quality of the participants of interest. The system firstly identifies the active conferees from the multiple incoming video streams by calculating the temporal and the spatial activities of the conferee sub-windows. The sub-windows of inactive participants are dropped and the saved bits are reallocated to the active sub-windows. Numerous motion vector composition schemes can be used to compose the unavailable motion vectors in the dropped frames due to limited bit-rates or frame-rates of the user clients in video transcoding. The present invention employs a pre-filtered activity-based forward dominant vector selection (PA-FDVS) scheme which provides accurate approximation of motion vectors with low computational cost and memory requirement. Simulation results show the visual quality of the active sub-windows is significantly improved at the cost of degrading the temporal resolution of the inactive sub-windows, which degradation is relatively invisible to human perception.

Claims

1. A video system, comprising:

(a) transcoder means for (i) receiving multiple incoming encoded digital video signals respectively sent over plural transmission paths from a plurality of video devices,

(ii) processing said received video signals and (iii) combining the processed video signals into an output video signal comprising a single coded video bit stream, respective portions of said output video signal corresponding to said video signals sent from said plurality of video devices constituting sub-windows of said output video signal; and

(b) means for transmitting the output video signal through said transmission paths to said plurality of video devices, sub-window;

 

wherein said transcoder means comprises;

(i) means for classifying said sub-windows into active sub-windows and static sub-windows; and

(ii) means for generating said output video signal by

(1) transcoding frames of said active sub-windows while skipping transcoding of frames of said static sub-windows and substituting a latest corresponding encoded sub-window for a skipped sub-window to approximate the skipped sub-window, and

(2) obtaining outgoing motion vectors of said output video signal from incoming motion vectors of said active sub-windows and said static sub-windows by obtaining a motion vector of a non-aligned macroblock, which is not aligned with segmented macroblock boundaries in said sub-windows, by a dominant vector selection operation comprising pre-filtering out unreliable motion vectors of said segmented macroblock boundaries and selecting the one of said segmented macroblock boundaries having the largest overlapping activity as the dominant block, and selecting the motion vector of said dominant block as said motion vector of said non-aligned macroblock.

4. The system of claim 1, wherein said means for classifying calculates a sum for each of said sub-windows of the magnitude of its motion vectors, compares the sum with a threshold, and classifies the sub-window as active or static in accordance with a comparison result.

5. The system of claim 1, wherein said means for processing obtains said outgoing motion vectors by, after a frame of a static sub-window is skipped, composing motion vectors of each skipped and non-skipped sub-window relative to its corresponding latest encoded sub-window.

6. The system of claim 1, wherein said dominant vector selection operation is a forward-dominant vector selection operation.

2. The system of claim 2, wherein said prefiltering operation comprises determining whether a strongly overlapping dominant block exists among said segmented macroblocks.

3. The system of claim 2, wherein:

(i) said operation of determining whether a strongly overlapping dominant block exists among said segmented macroblocks comprises;

calculating the largest overlapping area of each of the segmented macroblocks with said non-aligned macroblock, and if the largest overlapping area of one of the segmented macroblocks is greater than a predetermined threshold, then selecting the motion vector of the one of said segmented macroblocks with the largest overlapping area as the dominant vector, and if the largest overlapping area is not greater than said predetermined threshold, then setting an initial candidate list as the four motion vectors {IV₁, IV₂, IV₃, IV₄} of the segmented macroblocks, calculating the mean and the standard deviation of said four motion vectors in accordance with the relation;

${\mathrm{IV}}_{\mathrm{mean}}=\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{IV}}_i$${\mathrm{IV}}_{\mathrm{std}}=\sqrt{\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\left({\mathrm{IV}}_i-{\mathrm{IV}}_{\mathrm{mean}}\right)}^2}$

for i=1 to 4,

if |IV_i−

IV_mean|>

k_std·

IV_std, removing IV_i from the candidate list as unreliable, and if not, keeping IV_i in the candidate list as reliable,

(ii) said largest overlapping activity determining operation comprises, for each motion vector on the candidate list, calculating an area-activity product Ai·

ACTi, i=1,2,3,4, where Ai is the overlapping area with the segmented block (i) and ACTi is the activity measure, and

(iii) the dominant vector is selected as the motion vector of the one of said segmented macroblocks with the largest said area-activity product.

7. A video communication method comprising:

(a) receiving multiple incoming encoded digital video signals respectively sent over plural transmission paths from a plurality of video devices;

(b) processing said received video signals;

(c) combining the processed video signals into an output video signal comprising a single coded video bit stream; and

(d) transmitting the output video signal through said transmission paths to said plurality of video devices, respective portions of said output video signal corresponding to said video signals sent from said plurality of video devices constituting sub-windows of said output video signal;

 

wherein processing step (b) comprises;

(i) classifying said sub-windows into active sub-windows and static sub-windows; and

(ii) generating said output video signal by

(1) transcoding frames of said active sub-windows while skipping transcoding of frames of said static sub-windows and substituting a latest corresponding encoded sub-window for a skipped sub-window to approximate the skipped sub-window, and

(2) obtaining outgoing motion vectors of said output video signal from incoming motion vectors of said active sub-windows and said static sub-windows by obtaining a motion vector of a non-aligned macroblock which is not aligned with segmented macroblock boundaries in said sub-windows by a dominant vector selection operation comprising pre-filtering out unreliable motion vectors of said segmented macroblock boundaries and selecting the one of said segmented macroblock boundaries having the largest overlapping activity as the dominant block, and selecting the motion vector of said dominant block as said motion vector of said non-aligned macroblock.

8. The method of claim 7, wherein said prefiltering step comprises determining whether a strongly overlapping dominant block exists among said segmented macroblocks.

9. The method of claim 8, wherein:

(i) said step of determining whether a strongly overlapping dominant block exists among said segmented macroblocks comprises;

calculating the largest overlapping area of each of the segmented macroblocks with said non-aligned macroblock, and if the largest overlapping area of one of the segmented macroblocks is greater than a predetermined threshold, then selecting the motion vector of the one of said segmented macroblocks with the largest overlapping area as the dominant vector, and if the largest overlapping area is not greater than said predetermined threshold, then setting an initial candidate list as the four motion vectors {IV₁, IV₂, IV₃, IV₄} of the four segmented macroblocks, calculating the mean and the standard deviation of said four motion vectors in accordance with the relation;

${\mathrm{IV}}_{\mathrm{mean}}=\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{IV}}_i$${\mathrm{IV}}_{\mathrm{std}}=\sqrt{\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\left({\mathrm{IV}}_i-{\mathrm{IV}}_{\mathrm{mean}}\right)}^2}$

for i=1 to 4,

if |IV_i−

IV_mean|>

k_std·

IV_std, removing IV_i from the candidate list as unreliable, and if not, keeping IV_i in the candidate list as reliable;

(ii) said step of determining said largest overlapping activity comprises, for each motion vector on the candidate list, calculating an area-activity product Ai·

ACTi, i=1,2,3,4, where Ai is the overlapping area with the neighboring block (i) and ACTi is the activity measure, and

(iii) said method further comprises selecting the dominant vector as the motion vector of the one of said segmented macroblocks with the largest said area-activity product.

10. The method of claim 7, wherein said classifying step comprises calculating a sum for each of said sub-windows of the magnitude of its motion vectors, comparing the sum with a threshold, and classifying the sub-window as active or static in accordance with a comparison result.

11. The method of claim 7, wherein said processing step comprises obtaining said outgoing motion vectors by, after a frame of a static sub-window is skipped, composing motion vectors of each skipped and non-skipped sub-window relative to its corresponding latest encoded sub-window.

12. The method of claim 7, wherein said dominant vector selection operation is a forward dominant vector selection operation.

13. A video transcoder comprising:

(a) means for receiving multiple incoming encoded digital video signals respectively sent over plural transmission paths from a plurality of video devices,

(b) means for processing said received video signals and

(c) means for combining the processed video signals into an output video signal comprising a single coded video bit stream, respective portions of said output video signal corresponding to said video signals sent from said plurality of video devices constituting sub-windows of said output video signal;

 

wherein said means for processing comprises;

(i) means for classifying said sub-windows into active sub-windows and static sub-windows; and

(ii) means for generating said output video signal by

(1) transcoding frames of said active sub-windows while skipping transcoding of frames of said static sub-windows and substituting a latest corresponding encoded sub-window for a skipped sub-window to approximate the skipped sub-window, and

(2) obtaining outgoing motion vectors of said output video signal from incoming motion vectors of said active sub-windows and said static sub-windows by obtaining a motion vector of a non-aligned macroblock, which is not aligned with segmented macroblock boundaries in said sub-windows, by a dominant vector selection operation comprising pre-filtering out unreliable motion vectors of said segmented macroblock boundaries and selecting the one of said segmented macroblock boundaries having the largest overlapping activity as the dominant block, and selecting the motion vector of said dominant block as said motion vector of said non-aligned macroblock.

14. The transcoder of claim 13, wherein said prefiltering operation comprises determining whether a strongly overlapping dominant block exists among said segmented macroblocks.

15. The transcoder of claim 14, wherein:

(i) said operation of determining whether a strongly overlapping dominant block exists among said segmented macroblocks comprises;

calculating the largest overlapping area of each of the segmented macroblocks with said non-aligned macroblock, and if the largest overlapping area of one of the segmented macroblocks is greater than a predetermined threshold, then selecting the motion vector of the one of said segmented macroblocks with the largest overlapping area as the dominant vector, and if the largest overlapping area is not greater than said predetermined threshold, then setting an initial candidate list as the four motion vectors {IV₁, IV₂, IV₃, IV₄} of the segmented macroblocks, calculating the mean and the standard deviation of said four motion vectors in accordance with the relation;

${\mathrm{IV}}_{\mathrm{mean}}=\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{IV}}_i$${\mathrm{IV}}_{\mathrm{std}}=\sqrt{\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\left({\mathrm{IV}}_i-{\mathrm{IV}}_{\mathrm{mean}}\right)}^2}$

for i=1 to 4,

if |IV_i−

IV_mean|>

k_std·

IV_std, removing IV_i from the candidate list as unreliable, and if not, keeping IV_i in the candidate list as reliable,

(ii) said largest overlapping activity determining operation comprises, for each motion vector on the candidate list, calculating an area-activity product Ai·

ACTi, i=1,2,3,4, where Ai is the overlapping area with the segmented block (i) and ACTi is the activity measure, and

(iii) the dominant vector is selected as the motion vector of the one of said segmented macroblocks with the largest said area-activity product.

16. The transcoder of claim 13, wherein said means for classifying calculates a sum for each of said sub-windows of the magnitude of its motion vectors, compares the sum with a threshold, and classifies the sub-window as active or static in accordance with a comparison result.

17. The transcoder of claim 13, wherein said means for processing obtains said outgoing motion vectors by, after a frame of a static sub-window is skipped, composing motion vectors of each skipped and non-skipped sub-window relative to its corresponding latest encoded sub-window.

18. The transcoder of claim 13, wherein said dominant vector selection operation is a forward dominant vector selection operation.

19. A video transcoding method comprising:

(a) receiving multiple incoming encoded digital video signals respectively sent over plural transmission paths from a plurality of video devices;

(b) processing said received video signals; and

(c) combining the processed video signals into an output video signal comprising a single coded video bit stream, respective portions of said output video signal corresponding to said video signals sent from said plurality of video devices constituting sub-windows of said output video signal;

 

wherein processing step (b) comprises;

(i) classifying said sub-windows into active sub-windows and static sub-windows; and

(ii) generating said output video signal by

(1) transcoding frames of said active sub-windows while skipping transcoding of frames of said static sub-windows and substituting a latest corresponding encoded sub-window for a skipped sub-window to approximate the skipped sub-window, and

(2) obtaining outgoing motion vectors of said output video signal from incoming motion vectors of said active sub-windows and said static sub-windows by obtaining a motion vector of a non-aligned macroblock which is not aligned with segmented macroblock boundaries in said sub-windows by a dominant vector selection operation comprising pre-filtering out unreliable motion vectors of said segmented macroblock boundaries and selecting the one of said segmented macroblock boundaries having the largest overlapping activity as the dominant block, and selecting the motion vector of said dominant block as said motion vector-of said non-aligned macroblock.

20. The method of claim 19, wherein said prefiltering step comprises determining whether a strongly overlapping dominant block exists among said segmented macroblocks.

21. The method of claim 20, wherein:

(i) said step of determining whether a strongly overlapping dominant block exists among said segmented macroblocks comprises;

calculating the largest overlapping area of each of the segmented macroblocks with said non-aligned macroblock, and if the largest overlapping area of one of the segmented macroblocks is greater than a predetermined threshold, then selecting the motion vector of the one of said segmented macroblocks with the largest overlapping area as the dominant vector, and if the largest overlapping area is not greater than said predetermined threshold, then setting an initial candidate list as the four motion vectors {IV₁, IV₂, IV₃, IV₄} of the four segmented macroblocks, calculating the mean and the standard deviation of said four motion vectors in accordance with the relation;

${\mathrm{IV}}_{\mathrm{mean}}=\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{IV}}_i$${\mathrm{IV}}_{\mathrm{std}}=\sqrt{\frac{1}{4}<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\left({\mathrm{IV}}_i-{\mathrm{IV}}_{\mathrm{mean}}\right)}^2}$

for i=1 to 4,

if |IV_i−

IV_mean|>

k_std·

IV_std, removing IV_i from the candidate list as unreliable, and if not, keeping IV_i in the candidate list as reliable;

(ii) said step of determining said largest overlapping activity comprises, for each motion vector on the candidate list, calculating an area-activity product Ai·

ACTi, i=1,2,3,4, where Ai is the overlapping area with the neighboring block (i) and ACTi is the activity measure, and

(iii) said method further comprises selecting the dominant vector as the motion vector of the one of said segmented macroblocks with the largest said area-activity product.

22. The method of claim 19, wherein said classifying step comprises calculating a sum for each of said sub-windows of the magnitude of its motion vectors, comparing the sum with a threshold, and classifying the sub-window as active or static in accordance with a comparison result.

23. The method of claim 19, wherein said processing step comprises obtaining said outgoing motion vectors by, after a frame of a static sub-window is skipped, composing motion vectors of each skipped and non-skipped sub-window relative to its corresponding latest encoded sub-window.

24. The method of claim 19, wherein said dominant vector selection operation is a forward dominant vector selection operation.