Method and apparatus for scanning, personalizing, and casting multimedia data streams via a communication network and television

• US 20040111465A1
• Filed: 12/09/2002
• Published: 06/10/2004
• Est. Priority Date: 12/09/2002
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chuang, Wesley, Kaestle, Greg, Dao, Son, Vellaikal, Asha
• Granted Patent: US 7,870,279 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A streaming data filtering method and apparatus utilizing users'"'"' profiles to deliver streaming data to users is presented. The invention uses context-based cues to extract content from both video and text ontologies. The invention provides novel techniques that semi-automatically generate a media concept hierarchy using hierarchical classifiers for representing text, closed-caption, and video features. This hierarchy is used to classify arrival of real-time news and will also be used to match users'"'"' profiles. Moreover, this hierarchy can be dynamically re-organized based upon user changes and arrival of real-time news. Matching a request with respect to a concept hierarchy is more efficient and reliable than searching specific keywords since the views of collected documents are refined as the hierarchy is traversed.

Claims

1. An apparatus for scanning, personalizing, and casting multimedia data streams via a communication network comprising software modules operating on at least one data processing connected with a network of data processing systems, the software modules comprising:

a media matcher for classifying media streams based on media cues and for generating a media concept hierarchy based on the classifications;

a media repository for receiving and storing the media concept hierarchy, wherein the media concept hierarchy stores descriptive information regarding the classified media streams in the form of metadata;

a media planner for receiving user information and for generating a plan based on the user information for searching the metadata stored in the media concept hierarchy to locate media streams that match the user information; and

a media browser for receiving user information and for providing the user information to the media planner for use in generating the plan, and for receiving information regarding the media streams that match the user information to present the information to the user.

2. An apparatus as set forth in claim 1, further comprising a network of data processing systems in which the software modules reside.

3. An apparatus as set forth in claim 2, wherein the media matcher comprises at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

4. An apparatus as set forth in claim 3, wherein the operations of the media matcher, the media repository, the media planner, and the media browser are performed in real-time to enable classification of real-time data based on the user information.

5. An apparatus as set forth in claim 4, where the user information is a user profile.

6. An apparatus as set forth in claim 5, wherein the user profile is created by explicit user information input or by implicit user behavior.

7. An apparatus as set forth in claim 6, wherein the media matcher further comprises a media cues optimizer to use cues from a text classifier to trigger the video classifier in order to further enhance media stream classification accuracy.

8. An apparatus as set forth in claim 7, wherein the media matcher is configured to receive media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

9. An apparatus as set forth in claim 8, wherein the media planner locates media streams that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

10. An apparatus as set forth in claim 9, wherein the media browser operates on a user data processing system, and wherein the media planner, the media repository, and the media matcher operate on a data processing system remote from the user.

11. An apparatus as set forth in claim 10, wherein the data processing system on which the media planner, the media repository, and the media matcher operate selected from a group consisting of Internet service provider data processing systems, satellite earth terminal systems, and personal communication network servers.

12. An apparatus as set forth in claim 11, wherein the classifiers are based on term frequency inverse document frequency classifiers.

13. An apparatus as set forth in claim 12, wherein the text classifier is based on a distance metric.

14. An apparatus as set forth in claim 13, further comprising a semantic data network for providing metadata regarding media streams, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

15. An apparatus as set forth in claim 14, wherein the media browser further comprises a profile manager for storing profiles provided by users generated by the media planner to assist in locating media streams of interest, and a media streamer to enable streaming of media streams requested by users.

16. An apparatus as set forth in claim 15, wherein the media planner further comprises a semantic request parser for converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

17. An apparatus as set forth in claim 16, wherein the media planner further comprises an execution plan generator for generating a script, based on the user information, of how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

18. An apparatus as set forth in claim 8, wherein the media planner and the media browser operate on a user data processing system and wherein the media repository and media matcher operate on a data processing system remote from the user.

19. An apparatus as set forth in claim 1, further comprising a network of data processing systems in which the software modules reside.

20. An apparatus as set forth in claim 1, wherein the media matcher comprises at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

21. An apparatus as set forth in claim 1, wherein the operations of the media matcher, the media repository, the media planner, and the media browser are performed in real-time to enable classification of real-time data based on the user information.

22. An apparatus as set forth in claim 1, where the user information is a user profile.

23. An apparatus as set forth in claim 22, wherein the user profile is created by explicit user information input or by implicit user behavior.

24. An apparatus as set forth in claim 1, wherein the media matcher further comprises a media cues optimizer to use cues from a text classifier to trigger the video classifier in order to further enhance media stream classification accuracy.

25. An apparatus as set forth in claim 1, wherein the media matcher is configured to receive media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

26. An apparatus as set forth in claim 1, wherein the media planner locates media streams that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

27. An apparatus as set forth in claim 1, wherein the media browser operates on a user data processing system, and wherein the media planner, the media repository, and the media matcher operate on a data processing system remote from the user.

28. An apparatus as set forth in claim 1, wherein the data processing system on which the media planner, the media repository, and the media matcher operate selected from a group consisting of Internet service provider data processing systems, satellite earth terminal systems, and personal communication network servers.

29. An apparatus as set forth in claim 1, wherein the classifiers are based on term frequency inverse document frequency classifiers.

30. An apparatus as set forth in claim 1, wherein the text classifier is based on a distance metric.

31. An apparatus as set forth in claim 1, further comprising a semantic data network for providing metadata regarding media streams, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

32. An apparatus as set forth in claim 1, wherein the media browser further comprises a profile manager for storing profiles provided by users generated by the media planner to assist in locating media streams of interest, and a media streamer to enable streaming of media streams requested by users.

33. An apparatus as set forth in claim 1, wherein the media planner further comprises a semantic request parser for converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

34. An apparatus as set forth in claim 1, wherein the media planner further comprises an execution plan generator for generating a script, based on the user information, of how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

35. An apparatus as set forth in claim 1, wherein the media planner and the media browser operate on a user data processing system and wherein the media repository and media matcher operate on a data processing system remote from the user.

91. The apparatus as described in claim 1 wherein the media matcher further comprises:

i. a text classifier configured to classify closed captioned data into relevant nodes in the hierarchy

ii. a video classifier configured to classify video content into the concept hierarchy

92. The apparatus as described in claim 1 wherein the user profile is created by at least one of the following:

i. explicit user request

ii. implicit user request

93. The apparatus as described in claim 1 further comprising the media matcher that is configured to match the users request based on multimedia cues including at least one of the following:

i. spatial

ii. temporal or,

iii. user environment.

94. The apparatus as described in claim 1 which includes a means for discovering and retrieving multiple streaming data sources over at least one of the following:

i. an Internet data source

ii. a personal communications network

iii. a satellite based network

95. The apparatus as described in claim 1 where the media repository, the media planner, and the media matcher provide low-level data extraction and data classification at an uplink level of a satellite network;

a mobile satellite downlink station that directs streaming data from the streaming data sources, to the media browser and;

a fixed network including an additional media repository, an additional media planner, and an additional media matcher to provide higher level location, extraction, and classification requests.

96. The apparatus as described in claim 1 where existing closed-captioned data is used by the media matcher to provide the metadata for a media repository.

97. The apparatus as described in claim 1 further comprising a Semantic Data Network that is separate from the data sources and can support new content data and new request updates and represents semantic data based on at least one of the following:

i. classification techniques

ii. categorization techniques.

98. The apparatus as described in claim 1 further comprising a Real Time Media Casting architecture that defines processes for classifying incoming video and audio streams into a Semantic Data Network and for matching users profiles with the Semantic Data Network.

100. The method as described in claim 10 wherein the media matcher includes at least one of the following:

i. a text classifier configured to classify closed captioned data into relevant nodes in the hierarchy

ii. a video classifier configured to classify video content into the concept hierarchy

101. The method as described in claim 10 wherein the user specific profile is created by at least one of the following:

i. explicit user request

ii. implicit user request

102. The method as described in claim 10 wherein the media matcher uses multimedia cues including at least one of the following:

i. spatial

ii. temporal or

iii. user environment to match the user'"'"'s request.

103. The method as described in claim 10 which includes a method for discovering and retrieving multiple streaming data sources over at least one of the following:

i. an Internet data source

ii. a personal communications network

iii. a satellite based network

104. The method as described in claim 10 where:

the media repository, the media planner, and the media matcher function to extract and classify data at the uplink level of a satellite network;

the mobile satellite downlink station functions to direct the data to the media browser and;

a fixed network including an additional media repository, an additional media planner, and an additional media matcher function to provide higher level location, extraction, and classification requests.

105. The method as described in claim 10 where the media matcher uses existing closed-captioned data to provide the metadata to a media repository.

106. The method as described in claim 10 further utilizing a semantic data based classification and categorization technique that is separate from the data sources and supports new content and user request updates.

107. The method as described in claim 10 further defining processes for classifying incoming video and audio streams into a Semantic Data Network and for matching users profiles with the Semantic Data Network.

36. An method for scanning, personalizing, and casting multimedia data streams, the method comprising steps of:

classifying media streams based on media cues and generating a media concept hierarchy based on the classifications;

receiving and storing the media concept hierarchy, wherein the media concept hierarchy stores descriptive information regarding the classified media streams in the form of metadata;

receiving user information and generating a plan based on the user information for searching the metadata stored in the media concept hierarchy to locate media streams that match the user information; and

receiving user information and providing the user information for use in generating the plan, and receiving information regarding the media streams that match the user information to present the information to the user.

37. A method as set forth in claim 36, wherein the step of classifying media streams is performed by at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

38. A method as set forth in claim 37, wherein the steps in the method are performed in real-time to enable classification of real-time data based on the user information.

39. A method as set forth in claim 38, where the user information is a user profile.

40. A method as set forth in claim 39, wherein the user profile is created by explicit user information input or by implicit user behavior.

41. A method as set forth in claim 40, wherein the step of classifying media streams is performed by a media cues optimizer to use cues from a text classifier to trigger a video classifier in order to further enhance media stream classification accuracy.

42. A method as set forth in claim 41, wherein the step of classifying media streams is performed by receiving media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

43. A method as set forth in claim 42, wherein in the step of receiving user information and generating a plan, media streams are located that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

44. A method as set forth in claim 43, wherein the classifiers are based on term frequency inverse document frequency classifiers.

45. A method as set forth in claim 44, wherein the text classifier is based on a distance metric.

46. A method as set forth in claim 45, further comprising a step of providing metadata regarding media streams via a semantic data network, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

47. A method as set forth in claim 46, step of receiving user information and providing user information further comprises a step of storing profiles provided by users to assist in locating media streams of interest and streaming media streams requested by users via a media streamer.

48. A method as set forth in claim 47, wherein the step of receiving user information and generating a plan further comprises a step of converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

49. A method as set forth in claim 48, wherein step of receiving user information and for generating a plan further comprises a step of generating a script, based on the user information, regarding how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

50. A method as set forth in claim 36, wherein the step of classifying media streams is performed by at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

51. A method as set forth in claim 36, wherein the steps in the method are performed in real-time to enable classification of real-time data based on the user information.

52. A method as set forth in claim 36, where the user information is a user profile.

53. A method as set forth in claim 52, wherein the user profile is created by explicit user information input or by implicit user behavior.

54. A method as set forth in claim 36, wherein the step of classifying media streams is performed by a media cues optimizer to use cues from a text classifier to trigger a video classifier in order to further enhance media stream classification accuracy.

55. A method as set forth in claim 36, wherein the step of classifying media streams is performed by receiving media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

56. A method as set forth in claim 36, wherein in the step of receiving user information and generating a plan, media streams are located that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

57. A method as set forth in claim 36, wherein the classifiers are based on term frequency inverse document frequency classifiers.

58. A method as set forth in claim 36, wherein the text classifier is based on a distance metric.

59. A method as set forth in claim 36, further comprising a step of providing metadata regarding media streams via a semantic data network, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

60. A method as set forth in claim 36, step of receiving user information and providing user information further comprises a step of storing profiles provided by users to assist in locating media streams of interest and streaming media streams requested by users via a media streamer.

61. A method as set forth in claim 36, wherein the step of receiving user information and generating a plan further comprises a step of converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

62. A method as set forth in claim 36, wherein step of receiving user information and for generating a plan further comprises a step of generating a script, based on the user information, regarding how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

63. An computer program product for scanning, personalizing, and casting multimedia data streams, the computer program product comprising a computer-readable medium having means stored thereon for:

classifying media streams based on media cues and generating a media concept hierarchy based on the classifications;

receiving and storing the media concept hierarchy, wherein the media concept hierarchy stores descriptive information regarding the classified media streams in the form of metadata;

receiving user information and generating a plan based on the user information for searching the metadata stored in the media concept hierarchy to locate media streams that match the user information; and

receiving user information and providing the user information for use in generating the plan, and receiving information regarding the media streams that match the user information to present the information to the user.

64. A computer program product as set forth in claim 63, wherein the means for classifying media streams is performed by at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

65. A computer program product as set forth in claim 64, wherein the operations in the means of the computer program product are performed in real-time to enable classification of real-time data based on the user information.

66. A computer program product as set forth in claim 65, where the user information is a user profile.

67. A computer program product as set forth in claim 66, wherein the user profile is created by explicit user information input or by implicit user behavior.

68. A computer program product as set forth in claim 67, wherein the means for classifying media streams is performed by a media cues optimizer to use cues from a text classifier to trigger a video classifier in order to further enhance media stream classification accuracy.

69. A computer program product as set forth in claim 68, wherein the means for classifying media streams is performed by receiving media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

70. A computer program product as set forth in claim 69, wherein in the means for receiving user information and generating a plan, media streams are located that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

71. A computer program product as set forth in claim 70, wherein the classifiers are based on term frequency inverse document frequency classifiers.

72. A computer program product as set forth in claim 71, wherein the text classifier is based on a distance metric.

73. A computer program product as set forth in claim 72, further comprising a means for providing metadata regarding media streams via a semantic data network, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

74. A computer program product as set forth in claim 73, means for receiving user information and providing user information further comprises a means for storing profiles provided by users to assist in locating media streams of interest and streaming media streams requested by users via a media streamer.

75. A computer program product as set forth in claim 74, wherein the means for receiving user information and generating a plan further comprises a means for converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

76. A computer program product as set forth in claim 75, wherein means for receiving user information and for generating a plan further comprises a means for generating a script, based on the user information, regarding how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

77. A computer program product as set forth in claim 63, wherein the means for classifying media streams is performed by at least one classifier selected from a group consisting of text classifiers and video classifiers, wherein the text classifiers are configured for classifying closed-captioned data into the media concept hierarchy and the video classifiers are configured to classify video content into the concept hierarchy.

78. A computer program product as set forth in claim 63, wherein the operations of the means in the computer program product are performed in real-time to enable classification of real-time data based on the user information.

79. A computer program product as set forth in claim 63, where the user information is a user profile.

80. A computer program product as set forth in claim 79, wherein the user profile is created by explicit user information input or by implicit user behavior.

81. A computer program product as set forth in claim 63, wherein the means for classifying media streams is performed by a media cues optimizer to use cues from a text classifier to trigger a video classifier in order to further enhance media stream classification accuracy.

82. A computer program product as set forth in claim 63, wherein the means for classifying media streams is performed by receiving media streams from at least one of an Internet data source, a personal communication network, a digital video network, and a satellite-based network.

83. A computer program product as set forth in claim 63, wherein in the means for receiving user information and generating a plan, media streams are located that match the user information based on multimedia cues selected from a group consisting of spatial cues, temporal cues, and user environmental cues.

84. A computer program product as set forth in claim 63, wherein the classifiers are based on term frequency inverse document frequency classifiers.

85. A computer program product as set forth in claim 63, wherein the text classifier is based on a distance metric.

86. A computer program product as set forth in claim 63, further comprising a means for providing metadata regarding media streams via a semantic data network, where the semantic data network is external to media stream sources and is used to assist in classification and categorization of the media streams.

87. A computer program product as set forth in claim 63, means for receiving user information and providing user information further comprises a means for storing profiles provided by users to assist in locating media streams of interest and streaming media streams requested by users via a media streamer.

88. A computer program product as set forth in claim 63, wherein the means for receiving user information and generating a plan further comprises a means for converting the user information into a term frequency inverse document frequency vector for use in locating media streams that match the user information.

89. A computer program product as set forth in claim 63, wherein means for receiving user information and for generating a plan further comprises a means for generating a script, based on the user information, regarding how to locate and retrieve contents from different media streams classified in the media concept hierarchy.

90. A data parsing apparatus comprising:

a plurality of components including

i. media browser;

ii. a media repository comprising a storage apparatus for storing information related to a semantic segment of data mapped to a hierarchy index;

iii. a media planner comprising an apparatus that parses and matches a user specific profile with a Media Concept hierarchy that is configured to extract relevant metadata describing one or more data sources that have the contents matched with the users'"'"' request;

iv. a media matcher comprising an apparatus configured to classify data content into a concept hierarchy.

wherein said plurality of components are configured to locate, extract, and classify real time data based on user profile.

99. A data parsing method comprising:

a plurality of steps including;

i. receiving a user input for specific information;

ii. storing a semantic segment of data mapped to a hierarchy index on a media repository;

iii. parsing and matching a user specific profile with a hierarchy based software system that is configured to extract relevant metadata from one or more data sources and have the extracted data matched with the users'"'"' request on a media planner;

iv. using a media matcher to classify the data into a concept hierarchy.

wherein said plurality of steps locate, extract, and classify real time data based on a users'"'"' profile.