Video and digital multimedia aggregator remote content crawler

• US 20030028896A1
• Filed: 08/03/2001
• Published: 02/06/2003
• Est. Priority Date: 08/03/2001
• Status: Active Grant

• Current Assignee: Comcast IP Holdings I LLC (Comcast Corporation)
• Original Assignee: Comcast IP Holdings I LLC (Comcast Corporation)
• Inventors: Swart, William D., McCoskey, John S., Asmussen, Michael L.
• Granted Patent: US 8,285,701 B2
• Time in Patent Office: Days
• US Class Current: 725/127
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2543 Billing , e.g. for subscription services payment schemes, architectures or protocols G06Q20/00; e-commerce G06Q30/00; arrangements for billing for the use of broadcast information or broadcast-related information H04H60/21
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A remote content crawler continually crawls a digital communication network looking for content to provide to a content aggregator. The content provided to the aggregator may be stored in a form of an entire content file. The content may include an entire movie, television program or electronic book. Alternatively, the content provided to the aggregator may be a reference to a content file that is stored at, or that will be available at one of the remote locations. The content may be a reference to a future, scheduled live sports event that will be made available to system users. The sports event may be provided for a one time fee, as part of a sports package, for which a fee is collected, or as a free event.

Claims

1. A remote content crawler for use in a content search, packaging, and delivery system, comprising:

a remote content crawler processor that controls the remote content crawler;

a network resource processor that acquires data related to resources coupled to one or more communications networks;

a crawling criteria processor that acquires crawling criteria;

a crawler content provider processor that receives, processes and stores content provider listings; and

a network crawler, wherein the network crawler crawls content providers to acquire data related to available content.

2. The remote content crawler of claim 1, further comprising:

a content crawler results processor;

a metadata acquisition processor;

a plurality of crawling servers coupled to the network crawler; and

one or more databases, the one or more databases storing information and data generated in and received by the remote content crawler.

3. The remote content crawler of claim 2, wherein the one or more databases, comprises:

a content provider listing database;

a crawling criteria database; and

a network resources database.

4. An apparatus for searching one or more communications networks, accessing content available on the one or more communications networks, and acquiring access to the content, comprising:

one or more processors, wherein the one or more processors receive information related to the content; and

a network crawler coupled to the one or more processors, wherein the network crawler accesses the one or more communications networks to locate available content.

5. The apparatus of claim 4, wherein the network crawler comprises one or more crawling servers, wherein each of the one or more crawling servers searches the one or more communications networks according to a specific crawling criteria.

6. The apparatus of claim 5, wherein the network crawler is a World Wide Web robot, wherein the network crawler traverses a hypertext structure of the network and retrieves the content and recursively retrieves additional content referenced in the retrieved content.

7. The apparatus of claim 4, wherein the one or more processors, comprises:

a crawler processor coupled to the network crawler, wherein the crawler processor receives crawling schedule information and content search criteria;

a network resource processor coupled to the network crawler, wherein the network resource processor aggregates resource addresses of resources coupled to the one or more communications networks;

a crawling criteria processor that compiles data related to searches to be conducted by the network crawler and generates specific crawling criteria; and

a crawler content provider processor coupled to the network crawler that identifies, tracks, indexes and ranks providers of the content, and generates content provider data, wherein the network crawler receives the content provider data, the specific crawling criteria and the resource addresses and crawls the network based on the received content provider data, the specific crawling criteria, and the resource addresses.

8. The apparatus of claim 7, further comprising a content crawler results processor that receives content data from the network crawler, and that processes the content data and routes sorted and formatted crawling results for storage.

9. An apparatus for finding digital content in one or more communications networks, comprising:

means for building and maintaining network resource data, wherein the network resource data contains address data for content servers coupled to the one or more communications networks;

means, coupled to the means for building and maintaining network resource data, for storing the network resource data;

means for building and maintaining crawling criteria, wherein the crawling criteria are used during a crawling operation to search for the digital content;

means for building and maintaining content provider data, wherein the content provider data comprises data related to potential providers of content on the one or more communications networks; and

means, coupled to the means for building and maintaining network resource data, the means for building and maintaining crawling criteria, and the means for building and maintaining content provider data, for crawling the communications network.

10. The apparatus of claim 9, wherein the means for building and maintaining network resource data includes means for indexing address types.

11. The apparatus of claim 10, wherein the address types include top-level domain and subdomain names, Universal Resource Identifiers, Universal Resource Locators (URLs), and Internet Protocol (IP) address numbers.

12. The apparatus of claim 10, wherein the means for indexing address types is scalable to accommodate future naming conventions.

13. The apparatus of claim 9, wherein the means for building an maintaining the network resource data includes means for updating the address data.

14. The apparatus of claim 13, wherein the means for updating the address data, comprises:

means for receiving hyperlinked domain names;

means for downloading domain name records from public and private domain name registration databases;

means for synchronizing a local Domain Name Service (DNS) database with one or more DNS databases over the one or more communications networks;

means for performing reverse domain resolution by locating URLs associated with allowable IP addressing numbers; and

means for verifying Domain Name Service aliases and duplicate URLs against IP addresses to eliminate redundant domain names.

15. The apparatus of claim 9, wherein the network resource data comprises:

URL owner identity;

URL owner contact information;

available content types;

expiration time of the domain name; and

subdomain names to be excluded during crawling.

16. The apparatus of claim 9, wherein the crawling criteria, comprises:

terms, phrases and keywords;

data type descriptions;

metadata field names; and

metadata type descriptors, wherein the metadata type descriptors are associated with eligible content as one or more of hypertext descriptions and embedded file and data stream attributes and metadata.

17. The apparatus of claim 9, wherein the means for building and maintaining crawling criteria comprises automatic means for building and maintaining crawling criteria.

18. The apparatus of claim 17, wherein the automatic means comprises:

means for analyzing and importing metadata schemes for standardized and proprietary content formats;

means for parsing metadata field names and descriptive terms; and

means for analyzing hypertext associated with desired hyperlinks and for analyzing text proximate to the desired hyperlinks, wherein the means for analyzing hypertext identify terms that relate to a data type or content category.

19. The apparatus of claim 9, wherein the means for building and maintaining crawling criteria comprises manual means for building and maintaining crawling criteria.

20. The apparatus of claim 9, further comprising means for storing the crawling criteria.

21. The apparatus of claim 9, wherein the means for building and maintaining content provider data, comprises means for ranking content providers.

22. The apparatus of claim 21, wherein criteria for ranking the content providers, comprises:

quantity of available content;

provider professional association membership;

amount of content requested and downloaded by users of the communications network; and

content provider ratings, wherein the content provider ratings are provided by the users of the communications network.

23. The apparatus of claim 21, wherein a ranking of a content provider determines how frequently the content provider is crawled.

24. The apparatus of claim 9, wherein the means for crawling the communications network comprises one or more crawling servers, wherein the means for building and maintaining the network resource data comprises means for analyzing and subdividing the network resource data and means for providing the subdivided network resource data to the one or more crawling servers.

25. The apparatus of claim 24, wherein a crawling server comprises:

means for reading the subdivided network resource data;

means for communicating with a network resource; and

means for requesting and downloading data from the network resource.

26. The apparatus of claim 25, wherein the crawling server, further comprises:

means for comparing the content to the crawling criteria, wherein the crawling server provides data related to the content when the means for comparing indicates the crawling criteria are satisfied; and

means for following links from a first network resource to subsequent network resources, wherein the means for following links comprises;

means for analyzing hypertext structure of the first network resource to determine if the links have been crawled,

means for determining if a network resource has been downloaded or updated since a previous crawl of the network resource, and

means for analyzing the hypertext structure to determine if the link points to a network resource comprising a web page or other hypertext files.

27. The apparatus of claim 26, wherein the crawling server, further comprises:

means for caching hypertext files containing the data related to the content;

means for caching the links from the first network resource to subsequent network resources; and

means for indexing web pages and other hypertext files of interest.

28. The apparatus of claim 26, wherein the means for comparing the content to the crawling criteria comprises a comparison algorithm that compares elements in a hypertext file to the crawling criteria.

29. The apparatus of claim 9, further comprising:

means, coupled to the means for crawling the communications network, for acquiring and processing metadata related to a network resource; and

means, coupled to the means for acquiring and processing metadata related to a network resource, for processing content results from the crawled network resources.

30. A method for finding digital content in a communications network, comprising:

acquiring network resource data, wherein the network resource data comprises address data for content servers coupled to the one or more communications networks;

acquiring crawling criteria, wherein crawling criteria are used during a crawling operation to search for the digital content;

acquiring content provider data, wherein content provider data includes digital content provider-related data; and

crawling network resources in the one or more communications networks.

31. The method of claim 30, further comprising storing the network resource data, the crawling criteria, and the content provider data in one or more databases.

32. The method of claim 30, wherein acquiring network resource data comprises indexing the address data according to one or more address types.

33. The method of claim 32, wherein the address types include top-level domain and subdomain names, Universal Resource Identifiers, Universal Resource Locators (URLs), and Internet Protocol (IP) address numbers.

34. The method of claim 32, further comprising scaling the address types to accommodate future naming conventions.

35. The method of claim 30, further comprising updating the address data.

36. The method of claim 35, wherein updating the address data, comprises:

receiving hyperlinked domain names for the network resources;

downloading domain name records from public and private domain name registration sources;

synchronizing local Domain Name Service (DNS) databases with one or more DNS databases over the one or more communications networks;

performing reverse domain name resolution, comprising locating URLs associated with allowable IP address numbers;

verifying DNS aliases and duplicate URLs against IP addresses; and

eliminating any duplicate URLs identified by the verifying step.

37. The method of claim 30, wherein the network resource data comprises:

URL owner identity;

URL owner contact information;

available content types;

expiration time of the domain name; and

subdomain names to be excluded during crawling.

38. The apparatus of claim 30, wherein the crawling criteria, comprises:

terms, phrases and keywords;

data type descriptions;

metadata field names; and

metadata type descriptors, wherein the metadata type descriptors are associated with eligible content as one or more of hypertext descriptions and embedded file and data stream attributes and metadata.

39. The apparatus of claim 30, wherein acquiring the crawling criteria comprises automatically acquiring the crawling criteria.

40. The method of claim 39, wherein automatically acquiring the crawling criteria, comprises:

analyzing and importing metadata schemes for standardized and proprietary content formats;

parsing metadata field names and descriptive terms;

analyzing hypertext associated with desired hyperlinks;

analyzing text proximate to the desired hyperlinks, wherein analyzing hypertext identifies terms that relate to a data type or content category.

41. The method of claim 30, wherein acquiring the crawling criteria comprises acquiring the crawling criteria through manual input.

42. The method of claim 30, wherein acquiring the content provider data comprises ranking content providers.

43. The method of claim 42, wherein a ranking of a content provider is based on one or more of quantity of available content, provider professional association membership, amount of content requested and downloaded by users of the communications network, and content provider ratings, wherein the content provider ratings are provided by the users of the communications network.

44. The method of claim 43, further comprising determining a frequency of crawling a content provider based on the ranking of the content provider.

45. The method of claim 30, wherein crawling the network resources comprises crawling with one or more crawling servers.

46. The method of claim 45, farther comprising subdividing the network resources;

assigning the subdivided network resources to the one or more crawling servers; and

at a crawler server;

reading data from the assigned network resources,

communicating with the assigned network resources,

downloading data from the assigned network resources.

47. The method of claim 46, further comprising:

comparing digital content from one or more of the assigned network resources to the crawling criteria; and

acquiring data related to content that satisfies the crawling criteria.

48. The method of claim 46, further comprising:

following links from a first network resource to subsequent network resources, wherein following the links comprises;

analyzing hypertext structure of the first network resource to determine if the links have been crawled,

determining if a network resource has been downloaded or updated since a previous crawl of the network resource, and

analyzing the hypertext structure to determine if the link points to a network resource comprising a web page or other hypertext file.

49. The method of claim 48, further comprising:

caching hypertext files containing the data related to the content;

caching the links from the first network resource to subsequent network resources; and

indexing web pages or other hypertext files of interest.

50. The method of claim 48, wherein comparing the content to the crawling criteria comprises using a comparison algorithm that compares elements in a hypertext file to the crawling criteria.

51. The method of claim 30, further comprising:

acquiring and processing metadata related to a network resource; and

processing content results from the crawled network resources.

52. An apparatus for controlling a remote content crawler having one or more crawling servers, the remote content crawler capable of searching one or more communications networks for data related to content available on the one or more communications networks, the apparatus, comprising:

means for communicating with components of the one or more communications networks;

means, coupled to the communications means, for executing crawling of the one or more communications networks by the remote content crawler;

means, coupled to the means for executing crawling, for routing data received by the remote content crawler; and

means, coupled to the data routing means, for aggregating data related to resources of the one or more communications networks, wherein the remote content crawler uses the aggregated data to search the one or more communications networks.

53. The apparatus of claim 52, further comprising:

means, coupled to the communications means, for building a crawling criteria database, wherein the crawling criteria comprises one or more of hypertext search guidelines, data type list, metadata search criteria, and keyword lists.

54. The apparatus of claim 52, further comprising:

means for building a content provider database, wherein data related to content providers is tracked, indexed, and ranked.

55. The apparatus of claim 52, further comprising:

means for retrieving and routing metadata related to the content available on the one or more communications networks; and

means, coupled to the means for retrieving and routing the metadata related to the content available on the one or more communications networks, for indexing and formatting the retrieved metadata.

56. The apparatus of claim 52, wherein the means for executing crawling, comprises:

means for storing data related to crawling the one or more communications networks;

means for initiating crawling of the one or more communications networks, the means for initiating crawling comprising means for receiving administrative data related to the crawling of the one or more communications networks; and

means for analyzing a resource data set of the one or more communications networks to subdivide the resource dataset into one or more smaller resource data sets, wherein the subdivision is based on one or more of overall size of the resource data set, and a number of available crawling servers.

57. The apparatus of claim 57, wherein the means for executing crawling further comprises:

means for determining if contents of a hypertext files meet conditions of crawling criteria, comprising;

means for parsing the contents of the hypertext files, and

means for comparing the parsed content to the criteria in a criteria database, wherein if a hypertext file contains sufficient matching data, the hypertext file is cached.