System and process for broadcast and communication with very low bit-rate bi-level or sketch video

• US 20050041739A1
• Filed: 08/31/2004
• Published: 02/24/2005
• Est. Priority Date: 04/28/2001
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Gang, Li, Jiang, Shum, Heung-Yeung
• Granted Patent: US 7,916,794 B2
• Time in Patent Office: Days
• US Class Current: 375/240.120
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria

Abstract

A system and process for broadcast and communication with bi-level or sketch video at extremely low bandwidths is described. Essentially, bi-level and sketch video presents the outlines of the objects in a scene being depicted. Bi-level and sketch video provides a clearer shape, smoother motion, shorter initial latency and cheaper computational cost than do conventional DCT-based video compression methods. This is accomplished by converting each color or gray-scale image frame to bi-level or sketch image frame using adaptive thresholding method, compressing bi-level or sketch image frames into bi-level or sketch video using adaptive context-based arithmetic coding method. Bi-level or sketch video is particularly suitable to such small devices as Pocket PCs and mobile phones that possess small display screen, low bandwidth connection, and light computational power.

Claims

1. A computer-implemented process for generating a bi-level video, comprising using a computer to perform the following process actions:

inputting digitized video comprising a sequence of video image frames;

converting the input video image frames into bi-level image frames; and

encoding the bi-level image frames.

2. The process of claim 1, wherein the input video image frames have been compressed using a prescribed compression technique to facilitate storage or transmission, and wherein the process for generating the bi-level video further comprises an action of decompressing the input video image frames using a decompression technique appropriate for the prescribed compression technique, said decompression action being performed prior to the action of converting the input video image frames into bi-level image frames.

3. The process of claim 1, wherein the inputted digitized video further comprises an audio component associated with each image frame, and wherein the process for generating a bi-level video further comprises the actions of:

splitting the input video to separate the audio components from the image frames, prior to converting the image frames into bi-level image frames;

encoding the audio components using a low bit-rate audio compression technique;

synchronizing and combining each audio component with the bi-level image frame that was derived from the input video image frame associated with the audio component.

4. The process of claim 3, wherein the audio components have been compressed using a prescribed compression technique to facilitate storage or transmission, and wherein the process for generating the bi-level video further comprises an action of decompressing the audio components using a decompression technique appropriate for the prescribed compression technique, said decompression action being performed prior to encoding the audio components using the low bit-rate audio compression technique.

5. The process of claim 4, wherein the process actions of decompressing and encoding the audio components are performed concurrently with the process actions of converting the input video image frames into bi-level image frames and encoding the bi-level image frames.

6. The process of claim 1, wherein the digitized video is characterized by pixels defined in terms either a color level or a gray scale level, and wherein the process action of converting the input video image frames into bi-level image frames, comprises the actions of:

ascertaining whether the input video image frames comprises pixels defined in terms of a color level; and

converting each input video image frame so as to define each pixel thereof in terms of one of a plurality of gray scale levels so as to produce a gray scale image frame whenever the pixels are defined in terms of color levels, wherein the particular gray scale level selected to define a pixel is based on the color level of that pixel.

8. The process of claim 6, wherein the process action of converting the input video image frames into bi-level image frames, comprises, for each bi-level image frame, the actions of:

computing a threshold gray scale level;

assigning a first of two colors to those pixels that exhibit a gray scale level equal to or exceeding said threshold; and

assigning the second of said two colors to those pixel exhibiting a gray scale level that is less than said threshold.

24. The process of claim 1, wherein the process action of encoding the bi-level image frames, comprises an action of encoding the bi-level frames employing an adaptive context-based arithmetic coding technique.

25. The process of claim 1, wherein the process action of encoding the bi-level image frames, comprises the actions of, for each bi-level image frame:

encoding the bi-level frame employing an adaptive context-based arithmetic coding technique which utilizes a ½

probability table;

encoding the bi-level frame employing an adaptive context-based arithmetic coding technique which utilizes a pre-trained probability table;

determining whether the encoding utilizing a ½

probability table or the encoding utilizing a pre-trained probability table produces a bit stream with the lowest bit rate;

designating the bit stream produced using the probability table determined to produce the lowest bit rate as representing the encoded bi-level frame; and

incorporating in the bit stream designated as representing the encoded bi-level frame an indicator of the type of probability table used to encode the bit stream.

35. The process of claim 1, wherein the bi-level video is to be transmitted over a medium requiring a prescribed transmission encoding scheme, and wherein the process for generating a bi-level video further comprises a process action of packetizing the bi-level image frames in accordance with the prescribed packetizing scheme applicable to the medium over which the bi-level video is to be transmitted.

36. The process of claim 4, wherein the bi-level video is to be transmitted over a medium requiring a prescribed transmission packetizing scheme, and wherein the process for generating a bi-level video further comprises a process action of packetizing the combined bi-level frames and audio components in accordance with the prescribed packetizing scheme applicable to the medium over which the bi-level video is to be transmitted.

7. (cancelled)

9-23. -23. (cancelled)

26-34. -34. (cancelled)

37. A system for generating a bi-level video, comprising:

a video camera;

a sending device capable of transmitting encoded bi-level image frames and audio data and having a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input digitized video captured using said video camera, said video comprising a sequence of video image frames each with an associated audio component;

split the input video to separate the audio components from the image frames;

convert the input video image frames into bi-level image frames;

encode the bi-level image frames;

encode the audio components;

synchronize and combine each audio component with the bi-level image frame that was derived from the input video image frame associated with the audio component to produce a sequence of encoded bi-level video frames;

transmit each encoded bi-level video frame in sequence.

38. The system of claim 37, wherein the video camera is a digital video camera and the digitized video is input into the computing device directly from said camera through an appropriate interface.

39. The system of claim 37, wherein the video camera is a non-digital video camera and the images captured by the camera are digitized prior to being input into the computing device.

40. The system of claim 37, further comprising a receiving device which is capable of receiving and decoding bi-level video frames, displaying bi-level video image frames and playing audio data.

41. The system of claim 37, wherein the sending device is one of a (i) hand-held Personal Computer (PC), (ii) palm-sized PC, or (iii) mobile telephone.

42. A computer-readable medium having computer-executable instructions for generating a bi-level video, said computer-executable instructions comprising:

inputting digitized video comprising a sequence of video image frames;

converting the input video image frames into bi-level image frames; and

compressing the bi-level image frames using an adaptive context-based arithmetic coding technique.

43. The computer-readable medium of claim 42, further comprising an instruction for decompressing each input video image, prior to converting each image into a bi-level image, whenever the input video image frames are received in a compressed condition.

44. A computer-implemented process for playing a bi-level video, comprising using a computer to perform the following process actions:

receiving the bi-level video in the form of a sequence of frames each of which comprises a bi-level image;

decoding each bi-level image frame;

rendering each decoded bi-level image frame for display by assigning a first of two prescribed colors to pixel exhibiting a first of two binary values and the second of said prescribed colors to all remaining pixels of the image frame; and

displaying each rendered bi-level image frame using the assigned colors and in the sequence in which it was received.

45. The process of claim 44, wherein the first prescribed color is white and the second prescribed color is black.

46. The process of claim 44, wherein the second prescribed color is darker than the first prescribed color.

47. The process of claim 44, wherein the second prescribed color contrasts the first prescribed color to the extent that it is readily discernable by a person viewing the bi-level video.

48. The process of claim 44, wherein the bi-level video is transmitted over a medium requiring a prescribed transmission packetizing scheme, and wherein the process for playing a bi-level video further comprises a process action of unpacking the received frames of the b-level video using a unpacking procedure appropriate for the prescribed transmission encoding scheme.

49. The process of claim 44, wherein the bi-level video image frames have been encoded using an adaptive context-based arithmetic coding technique, and wherein the process action of decoding each bi-level image frame comprises using an adaptive context-based arithmetic decoding technique.

50. The process of claim 49, wherein the adaptive context-based arithmetic decoding technique comprises using a particular initial probability table for each bi-level image frame to predict whether each pixel in the image frame under consideration exhibits one or the other of the two bi-level image frame colors based on the values of a prescribed pattern of previously-predicted pixels, and wherein the process action of decoding the bi-level image frames, comprises the actions of:

receiving in conjunction with each encoded bi-level image frame a indicator specifying of the type of initial probability table that was used to encode that image frame; and

employing the indicated type of probability table as the initial probability table in the decoding of the bi-level image frame under consideration.

51. The process of claim 44, wherein the bi-level video image frames have been encoded using an adaptive context-based arithmetic coding technique, and wherein the process action of decoding each bi-level image frame comprises the actions of:

designating the first bi-level image frame received and frames in the frame sequence occurring at a known prescribed interval as I-frames;

designating the bi-level image frames in the frame sequence falling between a pair of consecutive I-frames, as p-frames;

decoding each I-frame using an adaptive context-based arithmetic decoding technique; and

decoding each p-frame using an adaptive context-based arithmetic decoding technique.

52. The process of claim 51, wherein the adaptive context-based arithmetic coding technique used to encode the bi-level image p-frames comprises determining the smallest bounding box that will surround all regions of dissimilarity in each p-frame, encoding just the portion of the p-frame contained within said bounding box, and incorporating in the bit stream designated as representing the encoded p-frame an indicator specifying the size and location of the bounding box in relation to an overall bi-level image frame, and wherein the process action of decoding each p-frame, comprises the actions of:

receiving in conjunction with each encoded p-frame the indicator specifying of the size and location of the bounding box in relation to the overall bi-level image frame; and

assuming all pixels located outside the bounding box in the overall image frame are have the same value as those in the corresponding locations in the last preceding frame.

53. The process of claim 44, wherein each bi-level video frame further comprises an audio data component, and wherein the process for playing a bi-level video further comprises the process actions of:

splitting each bi-level video frame into its constituent audio data component and bi-level image frame, prior to processing the image frame; and

playing the audio data associated with a bi-level video frame concurrently with performing the process action of displaying the bi-level image frame associated with that bi-level video frame.

54. The process of claim 53, wherein the audio data component of each bi-level video frame has been compressed using a audio compression technique, and wherein the process for playing a bi-level video further comprises a process action of decompressing the audio data associated with each frame of the bi-level video, prior to playing the audio data, using an audio decompression technique appropriate for the audio compression technique employed to compress the audio data component.

55. A system for playing a bi-level video, comprising:

a receiving device having a display capable of displaying bi-level images and playing audio data and having a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

receive the bi-level video in the form of a sequence of frames each of which comprises a bi-level image each bi-level video frame and an audio data component,

split each bi-level video frame into its constituent audio data component and bi-level image frame,

decode each bi-level image frame,

render each decoded bi-level image frame for display by assigning a first of two prescribed colors to pixel exhibiting a first of two binary values and the second of said prescribed colors to all remaining pixels of the image frame,

display each rendered bi-level image frame using the assigned colors and in the sequence in which it was received, and

play the audio data associated with each bi-level video frame concurrently with displaying the bi-level image frame associated with that bi-level video frame.

56. The system of claim 55, wherein the audio data component of each bi-level video frame has been compressed using an audio compression technique, and wherein the computer program further comprises a program module for decompressing the audio data associated with each frame of the bi-level video, prior to playing the audio data, using an audio decompression technique appropriate for the audio compression technique employed to compress the audio data component.

57. The system of claim 55, further comprising a sending device which is capable of sending audio data.

58. The system of claim 55, wherein the receiving device is one of a (i) hand-held Personal Computer (PC), (ii) palm-sized PC, or (iii) mobile telephone.

59. A computer-readable medium having computer-executable instructions for displaying a bi-level video, said computer-executable instructions comprising:

receiving the bi-level video in the form of a sequence of frames each of which comprises a bi-level image encoded using an adaptive context-based arithmetic coding technique;

decoding each bi-level image frame using an adaptive context-based arithmetic decoding technique;

rendering each decoded bi-level image frame for display by assigning a first of two prescribed colors to pixel exhibiting a first of two binary values and the second of said prescribed colors to all remaining pixels of the image frame; and

displaying each rendered bi-level image frame using the assigned colors and in the sequence in which it was received.