Method and apparatus for generating compact transcoding hints metadata

• US 20020157112A1
• Filed: 05/07/2002
• Published: 10/24/2002
• Est. Priority Date: 03/13/2000
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kuhn, Peter
• Granted Patent: US 7,738,550 B2
• Time in Patent Office: Days
• US Class Current: 725/113
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/234336 by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

An audio/video (or audiovisual, “A/V”) signal processing apparatus and method for extracting a compact representation of a multimedia description and transcoding hints metadata for transcoding between different (e.g., MPEG) compressed content representations, manipulating (e.g., MPEG compressed) bitstream parameters such as frame rate, bit rate, session size, quantization parameters, and picture coding type structure (e.g., group of pictures, or “GOP”), classifying A/V content, and retrieving multimedia information.

Claims

1. A video/audio signal processing method for processing supplied video/audio signals, comprising the steps of:

describing transcoding target bitstream parameters;

extracting transcoding hints metadata;

storing the transcoding hints metadata;

separating A/V material into segments;

associating the transcoding hints metadata to the separated A/V segments; and

transcoding the A/V material.

2. A video/audio signal processing method according to claim 1, wherein the step of describing the transcoding target bitstream parameters comprises the steps of:

defining a bit rate of a second bitstream of compressed images;

defining a size of pictures of the second bitstream of compressed images;

defining a number of frames per second of the second bitstream of compressed images;

defining an aspect ratio of a pel of the second bitstream of compressed images;

defining a color depth of each of the pel of the second bitstream of compressed images;

defining whether progressive format is used for the second bitstream of compressed images;

defining whether interlaced format is used for the second bitstream of compressed images;

defining whether frame pictures are used for the second bitstream of compressed images;

defining whether hold pictures are used for the second bitstream of compressed images; and

defining a compression method of the second bitstream of compressed images.

3. A video/audio signal processing method according to claim 2, wherein the step of describing the transcoding target bitstream parameters further comprises the step of defining employed compression standards as defined by MPEG (Moving Pictures Expert Group).

4. A video/audio signal processing method according to claim 2, wherein the step of describing the transcoding target bitstream parameters further comprises the step of defining employed compression standards as defined by ITU-T (International Telecommunications Union Technical Standards Group).

5. A video/audio signal processing method according to claim 1, wherein the step of extracting the transcoding hints metadata comprises the steps of:

receiving a first bitstream of compressed image data having a first GOP structure;

obtaining first motion information from the first bitstream;

obtaining texture/edge information of a first segmentation;

obtaining feature points and associated motion information from the first bitstream; and

obtaining region of interest information from the first bitstream.

6. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of storing the first motion information as transcoding hints metadata.

7. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing motion-related transcoding hints metadata as parameters of a parametric motion model.

8. A video/audio signal processing method according to claim 7, wherein the step of extracting the transcoding hints metadata further comprises the step of employing the parametric motion model to describe a global motion within subsequent rectangular video frames.

9. A video/audio signal processing method according to claim 7, wherein the step of extracting the transcoding hints metadata further comprises the step of employing the parametric motion model to describe a motion within a defined region of arbitrary shape.

10. A video/audio signal processing method according to claim 9, wherein the parametric motion model is employed to describe the motion within the defined region of arbitrary shape as used within MPEG-4.

11. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing motion-related transcoding hints metadata as an array of motion vectors contained in the first bitstream of the compressed image data.

12. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing motion-related transcoding hints metadata as an array of motion vectors derived from motion vectors contained in the first bitstream of the compressed image data.

13. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing motion-related transcoding hints metadata as a list of feature points with associated motion vectors, which are tracked within subsequent frames.

14. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing motion-related transcoding hints metadata as a list of feature points with associated motion vectors, which are tracked within arbitrarily shaped regions, within subsequent frames.

15. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing texture-related transcoding hints metadata as one of a list of DCT-coefficients and a measure (one of mean, minimum, maximum, variance, and standard deviation) derived thereof.

16. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing edge-related transcoding hints metadata as one of a list of DCT-coefficients and a measure (one of mean, minimum, maximum, variance, and standard deviation) derived thereof.

17. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing the feature points and associated motion-related transcoding hints metadata as a list.

18. A video/audio signal processing method according to claim 5, wherein the step of extracting the transcoding hints metadata further comprises the step of representing encoding-complexity-related transcoding hints metadata as a complexity metric derived from a life-time list of feature points tracked within subsequent frames by using a number of lost and new feature points from one frame to a next frame.

19. A video/audio signal processing method according to claim 1, wherein the step of storing the transcoding hints metadata comprises the step of maintaining a buffer containing transcoding hints metadata for several situations.

20. A video/audio signal processing method according to claim 19, wherein the step of storing the transcoding hints metadata further comprises the step of storing individual general transcoding hints metadata for several target devices.

21. A video/audio signal processing method according to claim 19, wherein the step of storing the transcoding hints metadata further comprises the step of storing general transcoding hints metadata for A/V segments of varying scene activity.

22. A video/audio signal processing method according to claim 1, wherein the step of separating the A/V material into segments comprises the steps of:

using feature points with associated motion vectors;

tracking the feature points and keeping a life-time of feature points; and

determining a new A/V segment for transcoding based on a number of feature points that could not be tracked from one frame to a next frame.

23. A video/audio signal processing method according to claim 1, wherein the step of associating the transcoding hints metadata to the separated A/V segments comprises the steps of:

calculating a number of new feature points per frame;

determining if the number of new feature points exceeds some thresholds; and

selecting based on said determination one of several transcoding hints states.

24. A video/audio signal processing method according to claim 1, wherein the step of transcoding the A/V material comprises the steps of:

receiving a first bitstream of compressed image data having a first GOP structure;

extracting transcoding hints metadata from the first bitstream;

utilizing the transcoding hints metadata associated to the first bitstream to facilitate transcoding; and

outputting a second bitstream.

25. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of utilizing the transcoding hints metadata associated to temporal segments of the first bitstream to facilitate transcoding.

26. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of utilizing the transcoding hints metadata associated to spatial segments of the first bitstream to facilitate transcoding.

27. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of utilizing motion information contained in the transcoding hints metadata to extrapolate second motion information for the second bitstream of compressed image data having a second GOP structure different from the first GOP structure.

28. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of controlling a bit rate of the second bitstream so that a bit rate of the first bitstream is different from the bit rate of the second bitstream.

29. A video/audio signal processing method according to claim 28, wherein the step of transcoding the A/V material further comprises the step of adjusting a size of pictures represented by the first bitstream so that pictures represented by the second bitstream exhibits a size different from the size of the pictures represented by the first bitstream.

30. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of adjusting a size of pictures represented by the first bitstream so that pictures represented by the second bitstream exhibit a size different from the size of the pictures represented by the first bitstream.

31. A video/audio signal processing method according to claim 30, wherein the step of transcoding the A/V material further comprises the step of encoding the pictures represented by the second bitstream as field pictures when the pictures represented by the first bitstream are encoded as frame pictures.

32. A video/audio signal processing method according to claim 30, wherein the step of transcoding the A/V material further comprises the step of encoding the pictures represented by the second bitstream as frame pictures when the pictures represented by the first bitstream are encoded as field pictures.

33. A video/audio signal processing method according to claim 30, wherein the step of transcoding the A/V material further comprises the step of interlacing the pictures represented by the first bitstream when the pictures represented by the first bitstream are received as a progressive sequence so that the pictures represented by the second bitstream are output as an interlaced sequence.

34. A video/audio signal processing method according to claim 30, wherein the step of transcoding the A/V material further comprises the step of de-interlacing the pictures represented by the first bitstream when the pictures represented by the first bitstream are received as an interlaced sequence so that pictures represented by the second bitstream are output as a progressive sequence.

35. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of encoding pictures represented by the second bitstream as field pictures when pictures represented by the first bitstream are encoded as frame pictures.

36. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of encoding pictures represented by the second bitstream as frame pictures when pictures represented by the first bitstream are encoded as field pictures

37. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of interlacing pictures represented by the first bitstream when pictures represented by the first bitstream are received as a progressive sequence so that pictures represented by the second bitstream are output as an interlaced sequence.

38. A video/audio signal processing method according to claim 24, wherein the step of transcoding the A/V material further comprises the step of de-interlacing pictures represented by the first bitstream when pictures represented by the first bitstream are received as an interlaced sequence so that pictures represented by the second bitstream are output as a progressive sequence.

39. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing pictures of a first size;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures of a second size different from the first size; and

outputting the second bitstream.

40. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing pictures defining an interlaced sequence;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures defining a progressive sequence; and

outputting the second bitstream.

41. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing pictures defining a progressive sequence;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures defining an interlaced sequence; and

outputting the second bitstream.

42. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing frame pictures;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing field pictures; and

outputting the second bitstream.

43. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing field pictures;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing frame pictures; and

outputting the second bitstream.

44. A transcoding method, comprising the steps of;

receiving a first bitstream of compressed image data representing a main image;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing a portion of the main image; and

outputting the second bitstream.

45. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data having a plurality of coding parameters including at least one of a GOP structure, a picture size, a bit rate, a frame picture format, a field picture format, a progressive sequence, and an interlaced sequence;

extracting first motion-related transcoding hints metadata from the first bitstream;

storing the first motion-related transcoding hints metadata;

utilizing the stored first motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data having a plurality of coding parameters such that one or more of the coding parameters of the second bitstream are different from the coding parameters of the first bitstream; and

outputting the second bitstream.

46. A transcoding method comprising the steps of:

receiving a first bitstream of compressed image data representing pictures of a first size;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures of a second size different from the first size; and

outputting the second bitstream.

47. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing pictures defining an interlaced sequence;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures defining a progressive sequence; and

outputting the second bitstream.

48. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing pictures defining a progressive sequence;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing pictures defining an interlaced sequence; and

outputting the second bitstream;

49. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing frame pictures;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing field pictures; and

outputting the second bitstream.

50. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing field pictures;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing frame pictures; and

outputting the second bitstream.

51. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data representing a main image;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data representing a portion of the main image; and

outputting the second bitstream.

52. A transcoding method, comprising the steps of:

receiving a first bitstream of compressed image data having a plurality of coding parameters including at least one of a GOP structure, a picture size, a bit rate, a frame picture format, a field picture format, a progressive sequence, and an interlaced sequence;

extracting first feature point motion-related transcoding hints metadata from the first bitstream;

storing the first feature point motion-related transcoding hints metadata;

utilizing the stored first feature point motion-related transcoding hints metadata to extrapolate second motion information for a second bitstream of compressed image data having a plurality of coding parameters such that one or more of the coding parameters of the second bitstream are different from the coding parameters of the first bitstream; and

outputting the second bitstream.

53. A video processing method for processing supplied video signals, comprising the steps of:

receiving a source video; and

classifying contents of the source video using one of motion metadata, texture/edge metadata, and feature points and associated motion metadata, including a number of new feature points per frame.

54. A video processing method according to claim 53, wherein said method is used for determining transcoding parameters settings of a transcode.

55. A video processing method according to claim 53, wherein said method is used for organizing audiovisual material based on the classification of the contents of the source video.

56. An apparatus for processing supplied video/audio signals, comprising:

a target buffer for storing at least one description of transcoding target bitstream parameters;

an extraction unit for extracting transcoding hints metadata based on the at least one description;

a buffer for storing the transcoding hints metadata;

a segmenting unit for separating A/V material into segments; and

a transcoding unit for associating the transcoding hints metadata to the separated A/V segments and transcoding the A/V material.

57. A transcoding apparatus, comprising:

an input for receiving a first bitstream of compressed image data representing pictures of a first size;

a transcoding hints metadata extraction unit for extracting transcoding hints metadata from the first bitstream;

a buffer for storing the transcoding hints metadata;

a processing unit for utilizing the stored transcoding hints metadata to extrapolate motion information for a second bitstream of compressed image data different from the first bitstream; and

an output for outputting the second bitstream.

58. An apparatus for processing supplied video signals, comprising:

an input for receiving a source video; and

a processor for classifying contents of the source video using one of motion metadata, texture/edge metadata, and feature points and associated motion metadata, including a number of new feature points per frame.