INFORMATION SENSORS FOR SENSING WEB DYNAMICS
First Claim
1. A method comprising:
- scanning, by one or more processors, a set of information sensors to determine that a running condition is met for executing at least one information sensor in the set of information sensors;
at least partly in response to a determination the running condition is met for the at least one information sensor, retrieving metadata associated with the at least one information sensor, the metadata including an update frequency and code to extract one or more data elements from a data source, the code being user-editable and providing predefined functions for at least extracting the one or more data elements from the data source;
running, by the one or more processors, the code to;
locate the data source,identify the one or more data elements within the data source,andperiodically extract the one or more data elements from the data source according to the update frequency; and
storing each extracted data element as a data point in a structured time series.
0 Assignments
0 Petitions
Accused Products
Abstract
Disclosed herein are techniques and systems for building “information sensors,” which are programmable “focused crawlers” that periodically discover, extract, analyze and aggregate structured information around a topic from the Web. A platform for building an information sensor allows a user to specify one or more data elements within a data source that the user desires to monitor, and an update frequency at which the data elements are to be extracted. Code may be generated based on the user specifications for creation and submission of the information sensor for storage in a database with metadata containing the code and update frequency. Once created, information sensors are scanned to check if running conditions are met, and if met, they may be executed by retrieving the metadata using a sensor identifier (ID). The code is executed to locate a data source, and periodically extract specified data elements therefrom to output structured time-series data.
-
Citations
20 Claims
-
1. A method comprising:
-
scanning, by one or more processors, a set of information sensors to determine that a running condition is met for executing at least one information sensor in the set of information sensors; at least partly in response to a determination the running condition is met for the at least one information sensor, retrieving metadata associated with the at least one information sensor, the metadata including an update frequency and code to extract one or more data elements from a data source, the code being user-editable and providing predefined functions for at least extracting the one or more data elements from the data source; running, by the one or more processors, the code to; locate the data source, identify the one or more data elements within the data source, and periodically extract the one or more data elements from the data source according to the update frequency; and storing each extracted data element as a data point in a structured time series. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A system for executing an information sensor, the system comprising:
-
one or more processors; one or more memories comprising; a sensor scheduler maintained in the one or more and executable by the one or more processors to periodically scan a set of information sensors to determine that a running condition is met for execution of at least one information sensor in the set of information sensors, the at least one information sensor having an identifier (ID); a sensor worker module maintained in the one or more memories and executable by the one or more processors to retrieve metadata associated with the ID and to assign a worker to the at least one information sensor to execute the information sensor, the metadata including an update frequency and code that is user-editable to provide predefined functions for at least extracting one or more data elements from a data source, the worker being configured to run the code to; locate the data source, identify the one or more data elements within the data source to be extracted, and periodically extract the one or more data elements according to the update frequency, and the sensor worker module being configured to store each extracted data element in a database in association with a time and a version number associated with each extracted data element. - View Dependent Claims (11, 12, 13)
-
-
14. A computer-readable medium storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:
-
receiving, from a user, a specification of; a data element within a data source that the user desires to monitor using an information sensor, and an update frequency at which the information sensor is to extract the data element from the data source, generating code configured to extract the data element from the data source according to the update frequency, the code being further editable by the user by providing predefined functions for at least extracting the data element from the data source; and creating the information sensor by storing the information sensor in a database along with metadata specifying the code and the update frequency. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification