Method and apparatus for an application crawler

• US 20060230011A1
• Filed: 11/22/2005
• Published: 10/12/2006
• Est. Priority Date: 11/22/2004
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Truveo Incorporated (Warner Bros. Discovery, Inc.)
• Inventors: Tuttle, Timothy D., Kocks, Peter F., Beguelin, Adam L.
• Granted Patent: US 7,584,194 B2
• Time in Patent Office: Days
• US Class Current: 706/62
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A computer-implemented method is provided for searching for files on the Internet. In one embodiment, the method may provide an application crawler that assembles and dynamically instantiates all components of a web page. The instantiated web application may then be analyzed to locate desired components on the web page. This may involve finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

Claims

1. A computer-implemented method comprising:

crawling and indexing an object model of multiple running, instantiated documents or applications.

2. The method of claim 1 wherein crawling comprises going from machine to machine to traverse at least one document or application on each machine.

3. The method of claim 2 wherein machines are located in different physical locations and are coupled to a computer network.

4. The method of claim 1 wherein crawling comprises:

following a tree of clickable items;

activating items in an automated manner substantially similar to the manner a human user would.

5. The method of claim 1 wherein crawling comprises following a seed list of pages or applications.

6. The method of claim 1 wherein crawling comprises following a seed list of pages or applications selected from at least one of the following:

a list of URLs, a list of application pointers, or an RSS feed.

7. The method of claim 1 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications.

8. The method of claim 1 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and recording the objects location and contents.

9. The method of claim 1 wherein indexing comprises:

loading the documents or applications;

instantiating the documents or applications;

traversing the object tree in the object model of the instantiated documents or applications.

10. The method of claim 1 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and recording the node in a database.

11. The method of claim 1 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and saving a pointer to the node in a database.

12. The method of claim 1 wherein indexing comprises saving a plurality of uniform resource locators (URLs) associated with media into a database.

13. The method of claim 1 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and simulating mouse, keyboard, or other user events.

14. The method of claim 13 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing the object tree.

15. The method of claim 13 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing an object tree of a new application or document displayed as a result the event.

16. The method of claim 1 wherein the object is any node or element of the object model of the instantiated documents or applications.

17. The method of claim 1 wherein the documents or applications being crawled include at least one of the following:

a word processing file, an Acrobat file, a spreadsheet, a browser application, a media player, a multimedia application, or a metadata header embedded in a media stream.

18. The method of claim 1 wherein the documents or applications being crawled include an operating system.

19. The method of claim 1 wherein the documents or applications being crawled include a file system.

20. The method of claim 1 wherein crawling comprises crawling the following types of networks:

intranet, single machine, or multiple apps on a single machine.

21. The method of claim 1 wherein crawling comprises crawling the Internet.

22. The method of claim 1 wherein crawling comprises crawling any device on a TCP/IP network.

23. The method of claim 1 wherein crawling comprises crawling any device on a public network.

24. The method of claim 1 wherein crawling comprises crawling any device on a private network.

25. The method of claim 1 wherein the applications or documents are fully instantiated.

26. The method of claim 1 wherein only a portion of the applications or documents are instantiated.

27. The method of claim 1 wherein indexing also instantiates video files.

28. The method of claim 1 wherein indexing also instantiates video streams.

29. The method of claim 1 further comprising adding data-query interfaces to software objects in the running instantiated documents or applications to allow for interface with an application crawler.

30. The method of claim 1 further comprising lowering security controls to allow access to the object model.

31. A computer-implemented method for searching for video files on a computer network, the method comprising:

crawling and indexing an object model of multiple running, instantiated documents or applications to locate video files.

32. The method of claim 31 wherein crawling comprises going from machine to machine to traverse at least one document or application on each machine.

33. The method of claim 32 wherein machines are located in different physical locations and are coupled to a computer network.

34. The method of claim 31 wherein crawling comprises:

following a tree of clickable items;

activating items in an automated manner substantially similar to the manner a human user would.

35. The method of claim 31 wherein crawling comprises following a seed list of pages or applications.

36. The method of claim 31 wherein crawling comprises following a seed list of pages or applications selected from at least one of the following:

a list of URLs, a list of application pointers, or an RSS feed.

37. The method of claim 31 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications.

38. The method of claim 31 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and recording the objects location and contents.

39. The method of claim 31 wherein indexing comprises:

loading the documents or applications;

instantiating the documents or applications;

traversing the object tree in the object model of the instantiated documents or applications.

40. The method of claim 31 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and recording the node in a database.

41. The method of claim 31 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and saving a pointer to the node in a database.

42. The method of claim 31 wherein indexing comprises saving a plurality of uniform resource locators (URLs) associated with media into a database.

43. The method of claim 31 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and simulating mouse, keyboard, or other user events.

44. The method of claim 43 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing the object tree.

45. The method of claim 43 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing an object tree of a new application or document displayed as a result of the event.

46. The method of claim 31 wherein the object is any node or element in the object model of the instantiated documents or applications.

47. The method of claim 31 wherein the documents or applications being crawled include at least one of the following:

a word processing file, an Acrobat file, a spreadsheet, a browser application, a media player, a multimedia application, or a metadata header embedded in a media stream.

48. The method of claim 31 wherein the documents or applications being crawled include an operating system.

49. The method of claim 31 wherein the documents or applications being crawled include a file system.

50. The method of claim 31 wherein crawling comprises crawling the following types of networks:

intranet, single machine, or multiple apps on a single machine.

51. The method of claim 31 wherein crawling comprises crawling the Internet.

52. The method of claim 31 wherein crawling comprises crawling any device on a TCP/IP network.

53. The method of claim 31 wherein crawling comprises crawling any device on a public network.

54. The method of claim 31 wherein crawling comprises crawling any device on a private network.

55. The method of claim 31 wherein the applications or documents are fully instantiated.

56. The method of claim 31 wherein only a portion of the applications or documents are instantiated.

57. The method of claim 31 wherein indexing also instantiates video files.

58. The method of claim 31 wherein indexing also instantiates video streams.

59. The method of claim 31 further comprising adding data-query interfaces to software objects in the running instantiated documents or applications to allow for interface with an application crawler.

60. The method of claim 31 further comprising lowering security controls to allow access to the object model.

61. A computer-implemented method for creating a searchable database, the method comprising:

crawling an object model of multiple running, instantiated documents or applications to locate video files;

indexing video files found in the object model by saving pointers to the video files in the database; and

extracting metadata about the video files from the object model and saving the metadata in the database.

62. A computer-implemented method for searching for files on the Internet, the method comprising:

providing a protocol crawler for identifying video-rich websites; and

providing an application crawler comprising;

an inspector for dynamically instantiating and assembling all components of a web page at one of said video-rich websites to create at least one instantiated web application;

an extractor for identifying specific parts of the instantiated web application that contain useful information and providing the logic required to extract that information into a metadata record; and

a crawler for analyzing the instantiated web application, finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

63. The method of claim 62 wherein the inspector includes code for software components for at least one of the following:

a Document Object Model (DOM) implementation for one or multiple browsers;

a scripting engine capable of executing JavaScript, JScript, ECMAScript or VBScript;

an XML parsing engine;

a Cascading Style Sheet engine;

a network I/O library;

an HTML parsing and rendering engine;

an engine for executing embedded controls such as ActiveX controls;

or

an engine for rendering web applications

64. The method of claim 62 wherein the extractor includes code for software components for at least one of the following:

an XSL engine;

an XPath implementation;

a regular expression engine;

a script execution engine;

an embedded object inspector for components such as, but not limited to, ActiveX and COM objects;

a network transport proxy, such as an http proxy

a rtsp or other multimedia stream proxy;

a software bridge to process data with class libraries of external programming frameworks;

a taxonomy engine for categorizing metadata;

or

a text parsing and processing engine.

65. The method of claim 62 wherein the crawler includes code for software components for at least one of the following:

a file I/O library;

a network I/O library;

or

a library for generating and storing logfiles.

66. A computer-implemented method for searching for files on the Internet, the method comprising:

finding a target URL;

downloading the HTML file for the target URL;

downloading supplementary data files used to build the complete web application, based on the information in the HTML file;

assembling application components from said supplementary data files and the HTML file;

instantiating application components to create a web application;

applying data-query interfaces to all objects in the web application that may contain useful data;

loading a pre-defined Application template or generating and automatically-defining a Application template;

applying the Application template to the extract all of the desired information from the web application.

saving the desired information to a file or database as a structured-data information record;

examining all components in the web application to identify all possible components that could respond to a mouse event or form a clickable item;

determining which clickable items have appeared since the last simulated mouse event;

storing new clickable items in an appropriate data structure, such as a new branch of a tree containing all clickable items in the application at all possible application states; and

simulating a mouse click on the first clickable item in the current branch of the clickable item tree.

67. The method of claim 66 further comprising repeating this method until the entire tree of clickable items has been traversed.

68. The method of claim 66 wherein instantiating step includes at least one of the following:

rendering HTML and constructing the Document Object Model;

applying style sheets;

executing scripts in a the appropriate script interpreter;

activating any controls or plug-ins, such as ActiveX controls;

launching video or audio streams;

launching animations such as Flash animations;

or

executing HTML behavior scripts.

69. A computer system comprising:

an application crawler having programming code for crawling and indexing an object model of running, instantiated documents or applications from the websites.

70. The computer system of claim 69 further comprising:

a protocol crawler for identifying websites for inspection wherein the application crawler crawls and indexes the object model of running, instantiated documents or applications from the websites.

71. The computer system of claim 69 wherein the application crawler comprises:

an inspector for dynamically instantiating and assembling all components of a web page at one of the websites to create at least one instantiated web application.

72. The computer system of claim 69 wherein the application crawler comprises:

an extractor for identifying specific parts of the instantiated web application that contain useful information and providing the logic required to extract that information into a metadata record.

73. The computer system of claim 69 wherein the application crawler comprises:

a crawler for analyzing the instantiated web application, finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

74. The computer system of claim 69 wherein the protocol crawler identifies websites for inspection that contain at least one video file.

75. The computer system of claim 69 wherein the protocol crawler identifies websites for inspection that contain at least one media file.

76. The computer system of claim 69 wherein the protocol crawler identifies websites for inspection that contain at least one of the following:

an mpeg file, an avi file, a real media file, a flash file, a .ts file, a .wmv file, or a Quicktime™

file.

77. The computer system of claim 69 wherein the application crawler is configured to inspect a tree of clickable items and activate each item in an automated manner substantially similar the manner a human user would.

78. The computer system of claim 69 wherein the application crawler uses a template configured for at least one of the following:

data extraction, timing of when to follow a link, depth to crawl, how to skip a commercial, where to start a crawl, finding links, location of title, location of media file metadata, temporal synchronization, or instructing the crawler to wait certain time intervals and then crawl the object tree again.

79. A system comprising:

a protocol crawler for identifying video-rich websites; and

an application crawler comprising;

an inspector for dynamically instantiating and assembling all components of a web page at one of said video-rich websites to create at least one instantiated web application;

an extractor for identifying specific parts of the instantiated web application that contain useful information and providing the logic required to extract that information into a metadata record; and

a crawler for analyzing the instantiated web application, finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

80. The computer system of claim 79 wherein the inspector includes code for software components for at least one of the following:

a Document Object Model (DOM) implementation for one or multiple browsers;

a scripting engine capable of executing JavaScript, JScript, ECMAScript or VBScript;

an XML parsing engine;

a Cascading Style Sheet engine;

a network I/O library;

an HTML parsing and rendering engine;

an engine for executing embedded controls such as ActiveX controls;

or

an engine for rendering web applications.

81. The computer system of claim 79 wherein the extractor includes code for software components for at least one of the following:

an XSL engine;

an XPath implementation;

a regular expression engine;

a script execution engine;

an embedded object inspector for components such as, but not limited to, ActiveX and COM objects;

a network transport proxy, such as an http proxy

a rtsp or other multimedia stream proxy;

a software bridge to process data with class libraries of external programming frameworks;

a taxonomy engine for categorizing metadata;

or

a text parsing and processing engine.

82. The computer system of claim 79 wherein the crawler includes code for software components for at least one of the following:

a file I/O library;

a network I/O library;

or

a library for generating and storing logfiles.

83. A computer implemented method comprising:

receiving a target URL; and

indexing an object model of multiple running, instantiated documents or applications.

84. The method of claim 83 wherein the receiving step comprises receiving a list of URLs.

85. The method of claim 83 wherein the receiving step comprises receiving at least one of the following:

a list of URLs, a list of application pointers, or an RSS feed.

86. The method of claim 83 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications.

87. The method of claim 83 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and recording the objects location and contents.

88. The method of claim 83 wherein indexing comprises:

loading the documents or applications;

instantiating the documents or applications;

traversing the object tree in the object model of the instantiated documents or applications.

89. The method of claim 83 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and recording the node in a database.

90. The method of claim 83 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and saving a pointer to the node in a database.

91. The method of claim 83 wherein indexing comprises saving a plurality of uniform resource locators (URLs) associated with media into a database.

92. The method of claim 83 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and simulating mouse, keyboard, or other user events.

93. The method of claim 92 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing the object tree.

94. The method of claim 92 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing an object tree of a new application or document displayed as a result the event.

95. The method of claim 83 wherein the object is any node or element of the object model of the instantiated documents or applications.

96. The method of claim 83 wherein the documents or applications being indexed include at least one of the following:

a word processing file, an Acrobat file, a spreadsheet, a browser application, a media player, a multimedia application, or a metadata header embedded in a media stream.

97. The method of claim 83 wherein the applications or documents are fully instantiated.

98. The method of claim 83 wherein only a portion of the applications or documents are instantiated.

99. The method of claim 83 wherein indexing also instantiates video files.

100. The method of claim 83 wherein indexing also instantiates video streams.

101. The method of claim 83 further comprising adding data-query interfaces to software objects in the running instantiated documents or applications to allow for interface with an application crawler.

102. The method of claim 83 further comprising lowering security controls to allow access to the object model.

103. A computer program product comprising:

a computer usable medium and computer readable code embodied on said computer usable medium, the computer readable code comprising;

computer readable program code configured to cause a computer to effect crawling and indexing of an object model of a running, instantiated document or application.

104. The product of claim 103 wherein crawling comprises going from machine to machine to traverse at least one document or application on each machine.

105. The product of claim 104 wherein machines are located in different physical locations and are coupled to a computer network.

106. The product of claim 103 wherein crawling comprises:

following a tree of clickable items;

activating items in an automated manner substantially similar to the manner a human user would.

107. The product of claim 103 wherein crawling comprises following a seed list of pages or applications.

108. The product of claim 103 wherein crawling comprises following a seed list of pages or applications selected from at least one of the following:

a list of URLs, a list of application pointers, or an RSS feed.

109. The product of claim 103 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications.

110. The product of claim 103 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and recording the objects location and contents.

111. The product of claim 103 wherein indexing comprises:

loading the documents or applications;

instantiating the documents or applications;

traversing the object tree in the object model of the instantiated documents or applications.

112. The product of claim 103 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and recording the node in a database.

113. The product of claim 103 wherein indexing comprises reaching into any node in the object model of the running, instantiated documents or application and saving a pointer to the node in a database.

114. The product of claim 103 wherein indexing comprises saving a plurality of uniform resource locators (URLs) associated with media into a database.

115. The product of claim 103 wherein indexing comprises traversing the object tree in the object model of the instantiated documents or applications and simulating mouse, keyboard, or other user events.

116. The product of claim 115 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing the object tree.

117. The product of claim 115 further comprising waiting a defined period of time after the mouse event, keyboard event, other user event, a browser event, or a mediaplayer event, and then traversing an object tree of a new application or document displayed as a result the event.

118. The product of claim 103 wherein the object is any node or element of the object model of the instantiated documents or applications.

119. The product of claim 103 wherein the documents or applications being crawled include at least one of the following:

a word processing file, an Acrobat file, a spreadsheet, a browser application, a media player, a multimedia application, or a metadata header embedded in a media stream.

120. The product of claim 103 wherein the documents or applications being crawled include an operating system.

121. The product of claim 103 wherein the documents or applications being crawled include a file system.

122. The product of claim 103 wherein crawling comprises crawling the following types of networks:

intranet, single machine, or multiple apps on a single machine.

123. The product of claim 103 wherein crawling comprises crawling the Internet.

124. The product of claim 103 wherein crawling comprises crawling any device on a TCP/IP network.

125. The product of claim 103 wherein crawling comprises crawling any device on a public network.

126. The product of claim 103 wherein crawling comprises crawling any device on a private network.

127. The product of claim 103 wherein the applications or documents are fully instantiated.

128. The product of claim 103 wherein only a portion of the applications or documents are instantiated.

129. The product of claim 103 wherein indexing also instantiates video files.

130. The product of claim 103 wherein indexing also instantiates video streams.

131. The product of claim 103 further comprising adding data-query interfaces to software objects in the running instantiated documents or applications to allow for interface with an application crawler.

132. The product of claim 103 further comprising lowering security controls to allow access to the object model.