Methods for transforming streaming video data

• US 7,339,993 B1
• Filed: 02/10/2000
• Issued: 03/04/2008
• Est. Priority Date: 10/01/1999
• Status: Expired due to Fees

• Current Assignee: Adaptive Streaming Inc. (Wi-LAN Inc.)
• Original Assignee: Vidiator Enterprises Incorporated
• Inventors: Singhal, Dave M., Molloy, Stephen A., Brooks, Roger K., Feng, Chi-Te, Zhang, Qing, Ma, Yanda
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): Czekaj, Dave
• Application Number:
  US09/502,549
• Time in Patent Office: 2,945 Days
• Field of Search: 375/240.26, 375/240.01, 709/233, 348/391.1, 725/74, 725/82, 725/86, 725/87, 725/91, 725/95, 725/97, 725/105, 725/115
• US Class Current: 375/240.26
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/440263 by altering the spatial resolution, e.g. for displaying on a connected PDA
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59

Abstract

A method for forming an output stream of data includes determining an output resolution for the output stream of data, determining an output frame rate for the output stream of data, determining an output color depth for the output stream of data, retrieving a first frame of data, a second frame of data, and a third frame of data from an input stream of data, the input stream of data having an input resolution, an input frame rate, and an input color depth, subsampling the first frame of data, the second frame of data, and the third frame of data to respectively form a first subsampled frame of data, a second subsampled frame of data, and a third subsampled frame of data, when the output resolution is lower than the input resolution, dropping the second subsampled frame of data, when the output frame rate is lower than the input frame rate, reducing color depth for the first subsampled frame of data and the second subsampled frame of data to respectively form a first reduced frame of data and a second reduced frame of data, when the output color depth is smaller than the input color depth, and converting the first reduced frame of data and the second reduced frame of data into the output stream of data.

Claims

1. A method for forming an output stream of data comprises:

determining an output resolution for the output stream of data;

determining an output frame rate for the output stream of data;

determining an output color depth for the output stream of data;

retrieving a first frame of data, a second frame of data, and a third frame of data from an input stream of data, the input stream of data having an input resolution, an input frame rate, and an input color depth;

subsampling the first frame of data, the second frame of data, and the third frame of data to respectively form a first subsampled frame of data, a second subsampled frame of data, and a third subsampled frame of data, if the output resolution is lower than the input resolution;

dropping the second subsampled frame of data, if the output frame rate is lower than the input frame rate;

reducing color depth for the first subsampled frame of data and the second subsampled frame of data to respectively form a first reduced frame of data and a second reduced frame of data, if the output color depth is smaller than the input color depth;

converting the first reduced frame of data and the second reduced frame of data into the output stream of data, wherein the subsampling, dropping, reducing, and converting are part of a transcoding process from among a plurality of transcoding processes, each of the transcoding processes being performed with said input stream of data, having same said input resolution, same said input frame rate, and same said input color depth, as their respective input; and

respectively generating multiple unique output streams of data from the plurality of transcoding processes and, at a server, selecting and separately sending concurrently the multiple unique output streams to respective multiple client devices, each of the multiple client devices being respectively sent their single unique output stream to form an image having a final output resolution without using additional streams to change the final output resolution of the image formed by that single unique output stream.

2. The method of claim 1, further comprising:

determining an output bit rate for each output stream of data; and

determining a scaling factor in response to the first reduced frame of data; and

scaling the second reduced frame of data by the scaling factor to change at least one of the output resolution, color depth, frame rate, and compression.

3. The method of claim 1 wherein each output stream of data is in a compressed format, the multiple unique output streams including:

at least one of;

MPEG-1, MPEG-2, MPEG-4; and

simultaneously at least one JPEG, GIF, WBMP, and another format different from an MPEG encoding format.

4. The method of claim 1 wherein the output streams of data are in a format including at least one of:

*.avi, *.rm, *.mov, and another encoding format different from an MPEG encoding format.

5. The method of claim 1 wherein the output resolution is a multiple of a frame having a resolution of 80 horizontal pixels by 60 vertical pixels.

6. The method of claim 1 wherein the output resolution is a multiple of 8 horizontal pixels.

7. The method of claim 1, further comprising:

cropping the first frame of data, the second frame of data, and the third frame of data before subsampling.

8. The method of claim 1 wherein at least one of the subsampling, dropping, reducing, and converting can be adapted dynamically based on either or both channel characteristics and client device characteristics.

9. The method of claim 1, further comprising wirelessly transmitting at least some of the multiple unique output streams of data.

10. The method of claim 1, further comprising determining at least one of the input format, input resolution, the input frame rate, and the input color depth from the input stream of data.

11. The method of claim 1 wherein each of the single unique output streams sent to the respective client devices have different final output resolutions from one another.

12. A method for converting an input video stream to an output video stream comprises:

deriving at least one of a specification of a resolution, a frame rate, a color depth, and encoding format for the output video stream;

deriving at least one of a specification of an input resolution, an input frame rate, an input color depth, and an input encoding format, for the input video stream;

obtaining a plurality of video frames from the input video stream;

subsampling each video frame from the plurality of video frames, if the resolution for the output video stream is different from the input resolution of the input video stream;

eliminating video frames from the plurality of video frames, if the frame rate for the output video stream is different from the input frame rate of the input video stream;

reducing color depth for video frames from the plurality of video frames, if the color depth for the output video stream is different from the input color depth of the input video stream;

converting the plurality of video frames to the output video stream in response to the encoding format for the output video stream, wherein at least some of the subsampling, eliminating, reducing, and converting is performed multiple times in respective multiple transcoding processes, with said input video stream having same said input resolution, same said input frame rate, same said input color depth, and same said input encoding format as respective input to each transcoding process, to simultaneously produce respective multiple different output video streams having different encoding formats; and

at a server, selecting and separately sending concurrently the multiple different output video streams to respective multiple client devices, each of the multiple client devices being respectively sent their single different output video stream to form an image having a final resolution without using additional streams to change the final resolution of that image formed by that single different output video stream.

13. The method of claim 12, further comprising:

deriving a specification of a bit rate for each output video stream; and

determining at least one type of scale factor for the plurality of video frames; and

wherein converting the plurality of video frames comprises converting the plurality of video frames to the output video streams in response to the encoding formats for the output video streams and in response to the scale factor, including changing the bit rate.

14. The method of claim 12 wherein the encoding formats for the output video streams comprises a standard compatible with the respective client devices.

15. The method of claim 12 wherein the encoding formats for the output video streams comprise a streaming format.

16. The method of claim 12 wherein the resolution for the output video streams is a multiple a resolution of 8 horizontal pixels by 8 vertical pixels.

17. The method of claim 12 wherein reducing color depth for video frames from the plurality of video frames occurs before subsampling each video frame from the plurality of video frames, and can be performed from any bit depth of any color format to any other bit depth of any other color format.

18. The method of claim 12, further comprising encrypting the output video streams.

19. The method of claim 12 wherein the input video stream includes color data and at least some of the output video streams includes only monochromatic data represented only by a Y component.

20. The method of claim 12 wherein the resolution for the output video streams is a multiple resolution of 8 horizontal pixels by 8 vertical pixels, if subsampling in a DCT domain.

21. The method of claim 12 wherein a horizontal resolution of a frame from the output video streams is a fractional multiple of a horizontal resolution of a video frame from the plurality of video frames from the input video stream.

22. The method of claim 12 wherein deriving the specification for the output video streams includes deriving this specification, including color format or video format, from information provided by a client device that is to receive their respective output video stream.

23. The method of claim 12 wherein deriving the specification for the input video stream includes deriving the information from the input video stream.

24. The method of claim 12 wherein at least one of the subsampling, eliminating, reducing, and converting can be adapted dynamically based on either or both channel characteristics and client device characteristics.

25. A method for operating a transcoder circuit coupled to a frame buffer comprises:

retrieving a plurality of frames of data from the frame buffer;

subsampling, with a sampling circuit in the transcoder circuit, the plurality of frames of data if the sampling circuit determines that an input resolution for the plurality of frames of data is different from a desired output resolution for the plurality of frames;

removing, with a frame rate circuit in the transcoder circuit, frames from the plurality of frames of data if the frame rate circuit determines that an input frame rate for the plurality of frames is different from a desired output frame rate for the plurality of frames;

reducing, with a color reduction circuit in the transcoder circuit, color depth from the plurality of frames of data if the color reduction circuit determines that an input color depth for the plurality of frames is different from a desired output color depth for the plurality of frames;

thereafter encoding, with an encoder circuit in the transcoder circuit, the plurality of frames of data into a desired output encoding format, wherein the transcoder circuit includes a plurality of transcoders to each receive the plurality of input frames of data having same said input resolution, input frame rate, and input color depth as their respective input, and to collectively output a corresponding plurality of unique output streams that have been generated using different subsampling, removing, reducing, and encoding performed by the transcoders; and

at a server, selecting and separately sending concurrently the plurality of unique output streams to respective multiple client devices, each of the multiple client devices being respectively sent their single unique output stream, which can have a different final output resolution relative to other single unique output streams sent to other client devices, without using additional streams to change that final output resolution.

26. The method of claim 25, further comprising outputting the plurality of frames of data in the desired output encoding format, at least some of the unique output streams having an encoding format different from an MPEG encoding format of other output streams.

27. The method of claim 25, further comprising scaling, with a scaling circuit in the transcoder circuit, at least one of an input bit rate, said input resolution, said input color depth, and a compression of the plurality of frames of data when the scaling circuit determines that the input bit rate for the plurality of frames is different from a desired output bit rate for the plurality of frames.

28. The method of claim 25 wherein subsampling the plurality of frames of data occurs before removing frames from the plurality of frames of data.

29. The method of claim 25 wherein sub sampling the plurality of frames of data occurs after removing frames from the plurality of frames of data.

30. The method of claim 25 wherein the encoder circuit is an MPEG-based encoding circuit.

31. The method of claim 25 wherein the encoder circuit is able to encode the plurality of frames of data into a plurality of different encoding formats, including formats different from an MPEG format.

32. The method of claim 25 wherein subsampling the plurality of frames of data comprises subsampling, with the sampling circuit in the transcoder circuit, the plurality of frames of data when the sampling circuit determines that said input resolution for the plurality of frames of data is higher than the desired output resolution for the plurality of frames.

33. The method of claim 25 wherein reducing color depth from the plurality of frames of data comprises reducing, with the color reduction circuit in the transcoder circuit, color depth from the plurality of frames of data when the color reduction circuit determines that said input color depth for the plurality of frames is greater than the desired output color depth for the plurality of frames.

34. The method of claim 25 wherein the encoder circuit is an M-JPEG encoding circuit.

35. The method of claim 25 wherein the encoder circuit is able to encrypt the plurality of frames of compressed data.

36. The method of claim 25, further comprising encrypting, with an encryption circuit in the transcoder circuit, the plurality of frames of data in the desired output encoding format.

37. The method of claim 25 wherein at least some of the subsampling, removing, reducing, and encoding can be adapted dynamically based on either or both channel characteristics and client device characteristics.

38. An article of manufacture, comprising:

a computer-readable medium storing thereon instructions that are executable by a computer to dynamically change characteristics of an input video stream to meet requirements for a plurality of output video streams, by;

determining input characteristics of an input stream of video data;

determining characteristics of a plurality of client devices;

determining characteristics of communication channels that can be used to send video data to the client devices;

adapting the input stream of video data into a plurality of different output streams of video data using a corresponding plurality of transcoding processes, including server-side instructions to determine which of each different output stream of video data is associated with and to be separately sent to respective client devices, wherein each output stream of video data has been adapted from the input stream of video data having same said input characteristics as common input into each respective transcoding process based on either or both characteristics of communication channels to or characteristics of respective client devices; and

dynamically updating, during transmission, at least some of the output streams in response to changes in characteristics of communication channels and characteristics of client devices that respectively receive the output streams, each of the plurality of client devices being respectively sent concurrently their single different output stream, which can have a different final output resolution relative to other single different output streams sent to other client devices, without using additional streams to change that final output resolution.

39. The article of manufacture of claim 38 wherein the instructions to adapt the input stream of video data into a plurality of different output streams of video data include instructions to provide different ones of at least some of these video data characteristics for each output stream:

spatial bandwidth, frame rate, color bandwidth, bit depth, frame rate, encoding format, color scheme, resolution, and bit rate.

40. The article of manufacture of claim 39 wherein the instructions to adapt the input stream of video data into a plurality of different output streams of video data include instructions to either increase or decrease at least one of the video data characteristics.

41. The article of manufacture of claim 38 wherein the instructions to determine input characteristics of the input stream of video data include instructions to determine input characteristics of either or both compressed and uncompressed video data in the input stream.

42. The article of manufacture of claim 38 wherein the instructions to adapt the input stream of video data into the plurality of different output streams of video data include instructions to simultaneously perform a plurality of different transcoding sessions, each transcoding session resulting in a respective generation of an output stream of video data.

43. A system, comprising:

a means for determining input characteristics of an input stream of video data;

a means for determining characteristics of a plurality of client devices;

a means for determining characteristics of communication channels that can be used to send video data to the client devices;

multiple transcoding means for adapting the input stream of video data, as common input with said input characteristics into the respective multiple transcoding means, into a respective plurality of different output streams of video data, including server-side means for determining which of each different output stream of video data is associated with and to be separately sent concurrently to respective client devices, wherein each output stream of video data has been adapted from the input stream of video data as common input with same said input characteristics based on either or both characteristics of communication channels to or characteristics of respective client devices; and

a means for dynamically updating, during transmission, at least some of the output streams in response to changes in characteristics of communication channels and characteristics of client devices that respectively receive the output streams, each of the plurality client devices being respectively sent concurrently their single different output stream, which can have a different final output resolution relative to other single different output streams sent to other client devices, without using additional streams to change that final output resolution.

44. The system of claim 43, further comprising additional means for processing the input stream of video data and for delivering the output streams of video data to their respective client devices.

45. The system of claim 44 wherein at least some of the means for delivering the output streams of video data include wireless means for delivering the output streams of video data to respective wireless client devices.