Method and apparatus for supporting advanced coding formats in media files

• US 20030163477A1
• Filed: 02/21/2003
• Published: 08/28/2003
• Est. Priority Date: 02/25/2002
• Status: Abandoned Application

• Current Assignee: Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Tabatabai, Ali, Visharam, Mohammed Zubair, Walker, Toby
• Application Number:
  US10/371,464
• Publication Number:
  US 20030163477A1
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/64792 Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8451 using Advanced Video Coding [AVC]
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

Sub-sample metadata defining sub-samples within each sample of multimedia data is created. Further, a file associated with the multimedia data is formed. This file includes the sub-sample metadata, as well as other information pertaining to the multimedia data.

Claims

1. A method comprising:

creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data; and

forming a file associated with the multimedia data, the file comprising the sub-sample metadata.

2. The method of claim 1 wherein each of the plurality of sub-samples is a sub-unit of a sample that may be decoded to obtain a partial reconstruction of the sample.

3. The method of claim 1 wherein creating sub-sample metadata comprises:

receiving a file with encoded multimedia data;

extracting information identifying boundaries of the plurality of sub-samples in the multimedia data; and

defining the sub-sample metadata based on the extracted information.

4. The method of claim 1 wherein creating sub-sample metadata comprises:

organizing the sub-sample metadata into a set of predefined data structures.

5. The method of claim 4 wherein creating sub-sample metadata further comprises:

converting each repeated sequence of data within the set of predefined data structures into a reference to a sequence occurrence and a number of occurrences.

6. The method of claim 4 wherein the set of predefined data structures comprises a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

7. The method of claim 1 further comprising:

sending the file associated with the multimedia data to a decoding system;

receiving the file associated with the multimedia data at the decoding system; and

extracting, at the decoding system, the sub-sample metadata from the file associated with the multimedia data, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples

8. A method comprising:

receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data; and

extracting the sub-sample metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples.

9. The method of claim 8 wherein each of the plurality of sub-samples is a sub-unit of a sample that may be decoded to obtain a partial reconstruction of the sample.

10. The method of claim 8 further comprising:

identifying the plurality of sub-samples within the multimedia file using the extracted sub-sample metadata; and

combining selected ones of the plurality of sub-samples into a packet to be sent to a media decoder.

11. The method of claim 8 wherein the extracted sub-sample metadata is organized into a set of predefined data structures.

12. The method of claim 11 wherein the set of predefined data structures comprises a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

13. A method comprising:

creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data; and

forming a file associated with the multimedia data, the file comprising the sub-sample metadata and the parameter set metadata.

14. The method of claim 13 wherein creating sub-sample metadata comprises:

organizing the sub-sample metadata into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

15. The method of claim 13 wherein each of the plurality of portions of multimedia data is any one of a sample and a sub-sample within the multimedia data.

16. The method of claim 13 wherein creating parameter set metadata comprises:

organizing the parameter set metadata into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of multimedia data.

17. A method comprising:

receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data and parameter set metadata identifying one or more parameter sets for the multimedia data; and

extracting the sub-sample metadata and the parameter set metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples and the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data.

18. The method of claim 17 wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data.

19. The method of claim 17 wherein the extracted parameter set metadata is organized into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of the multimedia data.

20. The method of claim 17 wherein the extracted sub-sample metadata is organized into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

21. A method comprising:

creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data;

creating sample group metadata defining groupings of a plurality of samples within the multimedia data; and

forming a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata and the sample group metadata.

22. The method of claim 21 wherein creating sub-sample metadata comprises:

organizing the sub-sample metadata into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

23. The method of claim 21 wherein each of the plurality of portions of multimedia data is any one of a sample and a sub-sample within the multimedia data.

24. The method of claim 21 wherein creating parameter set metadata comprises:

organizing the parameter set metadata into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of multimedia data.

25. The method of claim 21 wherein the groupings are based on inter-dependencies of the plurality of samples.

26. The method of claim 21 wherein creating sample group metadata comprises:

organizing the sample group metadata into a set of predefined data structures comprising a first data structure containing descriptive information about a plurality of sample groups within the multimedia data and a second data structure containing information that identifies samples in each of the plurality of sample groups.

27. A method comprising:

receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, and sample group metadata defining groupings of a plurality of samples within the multimedia data; and

extracting the sub-sample metadata, the parameter set metadata and the sample group metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples, the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data, and the extracted sample group metadata being subsequently used to identify samples that can be disposed of in future processing.

28. The method of claim 27 wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data.

29. The method of claim 27 wherein the extracted parameter set metadata is organized into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of the multimedia data.

30. The method of claim 27 wherein the extracted sub-sample metadata is organized into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

31. The method of claim 27 wherein the extracted sample group metadata is organized into a set of predefined data structures comprising a first data structure containing descriptive information about a plurality of sample groups within the multimedia data and a second data structure containing information that identifies samples in each of the plurality of sample groups.

32. A method comprising:

creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data;

creating sample group metadata defining groupings of a plurality of samples within the multimedia data;

creating switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and

forming a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata, the sample group metadata, and the switch sample metadata.

33. The method of claim 32 wherein creating sub-sample metadata comprises:

organizing the sub-sample metadata into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

34. The method of claim 32 wherein each of the plurality of portions of multimedia data is any one of a sample and a sub-sample within the multimedia data.

35. The method of claim 32 wherein creating parameter set metadata comprises:

organizing the parameter set metadata into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of multimedia data.

36. The method of claim 32 wherein the groupings are based on inter-dependencies of the plurality of samples.

37. The method of claim 32 wherein creating sample group metadata comprises:

organizing the sample group metadata into a set of predefined data structures comprising a first data structure containing descriptive information about a plurality of sample groups within the multimedia data and a second data structure containing information that identifies samples in each of the plurality of sample groups.

38. The method of claim 32 wherein each of the plurality of switch sample sets contains samples that have identical decoding values while using different reference samples.

39. The method of claim 32 wherein creating switch sample metadata comprises:

organizing the switch sample metadata into a predefined data structure represented as a table box containing a set of nested tables.

40. A method comprising:

receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, sample group metadata defining groupings of a plurality of samples within the multimedia data, and switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and

extracting the sub-sample metadata, the parameter set metadata, the sample group metadata and the switch sample metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples, the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data, the extracted sample group metadata being subsequently used to identify samples that can be disposed of in future processing, and the extracted switch sample metadata being subsequently used to find a replacement for a specific sample.

41. The method of claim 40 wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data.

42. The method of claim 40 wherein the extracted parameter set metadata is organized into a set of predefined data structures comprising a first data structure containing descriptive information about the one or more parameter sets and a second data structure containing information that defines associations between the one or more parameter sets and the plurality of portions of the multimedia data.

43. The method of claim 40 wherein the extracted sub-sample metadata is organized into a set of predefined data structures comprising a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

44. The method of claim 40 wherein the groupings are based on inter-dependencies of the plurality of samples.

45. The method of claim 40 wherein the extracted sample group metadata is organized into a set of predefined data structures comprising a first data structure containing descriptive information about a plurality of sample groups within the multimedia data and a second data structure containing information that identifies samples in each of the plurality of sample groups.

46. The method of claim 40 wherein each of the plurality of switch sample sets contains samples that have identical decoding values while using different reference samples.

47. The method of claim 40 wherein the extracted switch sample metadata is organized into a predefined data structure represented as a table box containing a set of nested tables.

48. A memory for storing data for access by an application program being executed on a data processing system, comprising:

a plurality of data structures stored in said memory, said plurality of data structures being resident in a file used by said application program, said file being associated with multimedia data and including sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data.

49. The memory of claim 48 wherein the file including the sub-sample metadata also includes the associated multimedia data.

50. The memory of claim 48 wherein the file including the sub-sample metadata contains references to a file containing the associated multimedia data.

51. The memory of claim 48 wherein the plurality of data structures comprises a first data structure containing information about sub-sample sizes, a second data structure containing information about a number of sub-samples in each sample, and a third data structure containing information describing each sub-sample.

52. A memory for storing data for access by an application program being executed on a data processing system, comprising:

a plurality of data structures stored in said memory, said plurality of data structures being resident in a file used by said application program, said file being associated with multimedia data and including

sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, and

parameter set metadata defining one or more parameter sets for a plurality of portions of the multimedia data.

53. A memory for storing data for access by an application program being executed on a data processing system, comprising:

a plurality of data structures stored in said memory, said plurality of data structures being resident in a file used by said application program, said file being associated with multimedia data and including

sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data,

parameter set metadata defining one or more parameter sets for a plurality of portions of the multimedia data, and

sample group metadata defining groupings of a plurality of samples within the multimedia data.

54. A memory for storing data for access by an application program being executed on a data processing system, comprising:

a plurality of data structures stored in said memory, said plurality of data structures being resident in a file used by said application program, said file being associated with multimedia data and including

sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data,

parameter set metadata defining one or more parameter sets for a plurality of portions of the multimedia data,

sample group metadata defining groupings of a plurality of samples within the multimedia data, and

switch sample metadata defining a plurality of switch sample sets associated with the multimedia data.

55. An apparatus comprising:

a metadata generator to create sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data; and

a file creator to form a file associated with the multimedia data, the file comprising the sub-sample metadata.

56. The apparatus of claim 55 wherein each of the plurality of sub-samples is a sub-unit of a sample that may be decoded to obtain a partial reconstruction of the sample.

57. The apparatus of claim 55 wherein the metadata generator is to create sub-sample metadata by receiving a file with encoded multimedia data, extracting information identifying boundaries of the plurality of sub-samples in the multimedia data, and defining the sub-sample metadata based on the extracted information.

58. An apparatus comprising:

a metadata extractor to receive a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, and to extract the sub-sample metadata from the file; and

a media data stream processor to utilize the extracted sub-sample metadata for accessing any of the plurality of sub-samples.

59. The apparatus of claim 58 wherein each of the plurality of sub-samples is a sub-unit of a sample that may be decoded to obtain a partial reconstruction of the sample.

60. The apparatus of claim 58 wherein the media data stream processor is further to identify the plurality of sub-samples within the multimedia file using the extracted sub-sample metadata, and to combine selected ones of the plurality of sub-samples into a packet to be sent to a media decoder.

61. An apparatus comprising:

a metadata generator to create sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data and to create parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data; and

a file creator to form a file associated with the multimedia data, the file comprising the sub-sample metadata and the parameter set metadata.

62. An apparatus comprising:

a metadata extractor to receive a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data and parameter set metadata identifying one or more parameter sets for the multimedia data, and to extract the sub-sample metadata and the parameter set metadata from the file; and

a media data stream processor to utilize the extracted sub-sample metadata for accessing any of the plurality of sub-samples and to utilize the extracted parameter set metadata for determining relationships between the one or more parameter sets and a plurality of portions of the multimedia data.

63. An apparatus comprising:

a metadata generator to create sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data, to create parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data, and to create sample group metadata defining groupings of a plurality of samples within the multimedia data; and

a file creator to form a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata and the sample group metadata.

64. An apparatus comprising:

a metadata extractor to receive a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, and sample group metadata defining groupings of a plurality of samples within the multimedia data, and to extract the sub-sample metadata, the parameter set metadata and the sample group metadata from the file; and

a media data stream processor to utilize the extracted sub-sample metadata for accessing any of the plurality of sub-samples, to utilize the extracted parameter set metadata for determining relationships between the one or more parameter sets and a plurality of portions of the multimedia data, and to utilize the extracted sample group metadata for identifying samples that can be disposed of in future processing.

65. An apparatus comprising:

a metadata generator to create sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data, to create parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data, to create sample group metadata defining groupings of a plurality of samples within the multimedia data, and to create switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and

a file creator to form a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata, the sample group metadata, and the switch sample metadata.

66. An apparatus comprising:

a metadata extractor to receive a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, sample group metadata defining groupings of a plurality of samples within the multimedia data and switch sample metadata defining a plurality of switch sample sets associated with the multimedia data, and to extract the sub-sample metadata, the parameter set metadata, the sample group metadata and the switch sample metadata from the file; and

a media data stream processor to utilize the extracted sub-sample metadata for accessing any of the plurality of sub-samples, to utilize the extracted parameter set metadata for determining relationships between the one or more parameter sets and a plurality of portions of the multimedia data, to utilize the extracted sample group metadata for identifying samples that can be disposed of in future processing, and to utilize the extracted switch sample metadata for finding a replacement for a specific sample.

67. An apparatus comprising:

means for creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data; and

means for forming a file associated with the multimedia data, the file comprising the sub-sample metadata.

68. An apparatus comprising:

means for receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data; and

means for extracting the sub-sample metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples.

69. An apparatus comprising:

means for creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

means for creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data; and

means for forming a file associated with the multimedia data, the file comprising the sub-sample metadata and the parameter set metadata.

70. An apparatus comprising:

means for receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data and parameter set metadata identifying one or more parameter sets for the multimedia data; and

means for extracting the sub-sample metadata and the parameter set metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples and the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data.

71. An apparatus comprising:

means for creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

means for creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data;

means for creating sample group metadata defining groupings of a plurality of samples within the multimedia data; and

means for forming a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata and the sample group metadata.

72. An apparatus comprising:

means for receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, and sample group metadata defining groupings of a plurality of samples within the multimedia data; and

means for extracting the sub-sample metadata, the parameter set metadata and the sample group metadata from the file, the extracted sub-sample metadata being subsequently used to access any of the plurality of sub-samples, the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data, and the extracted sample group metadata being subsequently used to identify samples that can be disposed of in future processing.

73. An apparatus comprising:

means for creating sub-sample metadata defining a plurality of sub-samples within each sample of multimedia data;

means for creating parameter set metadata identifying one or more parameter sets for a plurality of portions of the multimedia data;

means for creating sample group metadata defining groupings of a plurality of samples within the multimedia data;

means for creating switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and

means for forming a file associated with the multimedia data, the file comprising the sub-sample metadata, the parameter set metadata, the sample group metadata, and the switch sample metadata.

74. An apparatus comprising:

means for receiving a file associated with multimedia data, the file comprising sub-sample metadata defining a plurality of sub-samples within each sample of the multimedia data, parameter set metadata identifying one or more parameter sets for the multimedia data, sample group metadata defining groupings of a plurality of samples within the multimedia data and switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and

means for extracting the sub-sample metadata, the parameter set metadata, the sample group metadata and the switch sample metadata from the file, the sub-sample metadata being subsequently used to access any of the plurality of sub-samples, the extracted parameter set metadata being subsequently used to determine relationships between the one or more parameter sets and a plurality of portions of the multimedia data, the extracted sample group metadata being subsequently used to identify samples that can be disposed of in future processing, and the extracted switch sample metadata being subsequently used to find a replacement for a specific sample.