Web-based collaborative data collection system

• US 7,076,521 B2
• Filed: 06/15/2001
• Issued: 07/11/2006
• Est. Priority Date: 06/26/2000
• Status: Active Grant

• Current Assignee: Vertical Computer Systems Incorporated
• Original Assignee: Vertical Computer Systems Incorporated
• Inventors: Davison, Jeff
• Primary Examiner(s): Meky, Moustafa M.
• Application Number:
  US09/882,494
• Publication Number:
  US 20030101238A1
• Time in Patent Office: 1,852 Days
• Field of Search: 709200-203, 709217-224, 709/246
• US Class Current: 709/203
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation

Abstract

The present invention is directed to a system for gathering data from a web-based server, transmitting the data to a web-based client, and storing the data on the web-based client. The web-based server translates data from a data supplier'"'"'s proprietary data model into a data consumer'"'"'s proprietary data model using a data mapping function. The web-based server also converts data from a structured data format to a markup language format. The web-based client periodically polls one or more data servers for data. The web-based client receives data in a markup language format and translates it into a structured data format, then stores it in a database. The web-based client and the web-based server can collaborate with each other to streamline the data conversion and translation process.

Claims

1. A web-based client system for acquiring data from a web-enabled data server, comprising:

a web-based data collector, the data collector for collecting data from the web-enabled data server, the data collector for collecting data periodically, the data collector for determining the web-enabled data server to collect data from by referencing data source identifying information, the data collector for converting the data from a markup language format to a structured data storage format, the data collector for storing the data in a storage system.

2. The web-based client system of claim 1, wherein the data collector is a first data collector

and the web-enabled data server is a first web-enabled data server, further comprising a

second web-enabled data server, the second web-enabled data server for providing data from

the storage system to a second data collector.

3. The web-based client system of claim 1, wherein the markup language format comprises Extensible Markup Language (XML).

4. The web-based client system of claim 1, wherein the markup language format comprises Hypertext Markup Language (HTML).

5. The web-based client system of claim 1, wherein the structured data format comprises Structured Query Language (SQL).

6. The web-based client system of claim 1, wherein the data source identifying information is received from the web-enabled data server.

7. The web-based client system of claim 1, wherein the data source identifying information is received from a user.

8. The web-based client system of claim 1, wherein the data source identifying information comprises a Uniform Resource Locator (URL).

9. The web-based client system of claim 1, wherein the data collector includes a scheduler, the scheduler for setting up a data collection schedule.

10. The web-based client system of claim 1, wherein the data collector is adapted to step through a list of URLs and collects the content pointed to by the URLs.

11. The web-based client system of claim 1, wherein the data collector is adapted to convert the markup language format to the structured data storage format by using a mapping function,

the mapping function mapping markup language format data items to structured data storage format data items.

12. The web-based client system of claim 11, wherein the mapping function is adapted to map XML tags to SQL fields.

13. The web-based client system of claim 11, wherein the mapping function is adapted to map XML tags to database records.

14. The web-based client system of claim 11, wherein the mapping function comprises a first mapping function, further comprising a second mapping function, wherein the data collector is adapted to select the first mapping function based upon information contained in the data source identifying information.

15. The web-based client system of claim 1, wherein the data collector is adapted to store the data in the data storage system by inserting a new data item when the new data item appears in the data collected from the web-enabled data server, updating an existing data item content when a value of the existing data item in the data collected from the web-enabled data server changes, or deleting an old data item when the old data item is removed from the data collected from the web-enabled data server.

16. The web-based client system of claim 15, wherein the system is adapted to trigger an alarm when the content of an existing data item changes or when a new data item is added, or when an old data item is deleted.

17. The web-based client system of claim 16, wherein the system is adapted to take an action when the alarm is triggered.

18. The web-based client system of claim 1, wherein the system is written in an open scripting language, the open scripting language permitting publication of scripts.

19. The web-based client system of claim 1, wherein the system is adapted to be continuously running.

20. The web-based client system of claim 1, further comprising a web-based data processor, the data processor comprising a toolkit, the toolkit comprising a plurality of tools, each tool for performing a data processing operation on the storage system, the tools for processing the data stored in the storage system, the tools for presenting the data to a data user.

21. The web-based client system of claim 1, further comprising a web-based administrator module, the administrator module for configuring and maintaining the web-based client system, the administrator module comprising one or more web-based tools.

22. The web-based client system of claim 21, wherein the administrator module comprises a tool adapted to configure a list of URLs.

23. The web-based client system of claim 21, wherein the administrator module comprises a tool adapted to allow mapping of markup language format items to structured data storage format items.

24. The web-based client system of claim 23, wherein the markup language format items are XML tags and the structured data storage format items are database items in a relational database.

25. The web-based client system of claim 23, wherein the markup language format items are HTML tags and the structured data storage format items are database items in a relational database.

26. The web-based client system of claim 1, further comprising a web-enabled server, the server for converting structured data format data into Markup language format data and transmitting markup language data to the web-enabled client.

27. The web-based client system of claim 26, wherein the web-based client system is adapted to store data using a first data model, and the web based server system is adapted to store data in a second data model, different from the first data model.

28. A web-based data server system for transmitting data to a web-enabled client, comprising:

a web enabler agent, the web enabler agent for reading the data from a structured data formatted data storage system, the web enabler agent for converting the data from a structured data format to a markup language format, the web enabler agent for transmitting the data to a web-enabled client and;

a web-based data collector, the data collector for collecting data from a web-enabled data server, the data collector for collecting data periodically, the data collector for determining the web-enabled data server to collect data from by referencing data source identifying information, the data collector for converting the data from a markup language format to a structured data storage format, the data collector for storing the data in a storage system.

29. The web-based data server system of claim 28, wherein the markup language format comprises Extensible Markup Language (XML).

30. The web-based data server system of claim 28, wherein the markup language format comprises Hypertext Markup Language (HTML).

31. The web-based data server system of claim 28, wherein the structured data format comprises Structured Query Language (SQL).

32. The web-based data server system of claim 28, wherein the data collector is adapted to convert the structured data storage format to the markup language format by using a mapping function, the mapping function for mapping structured data storage format data items to markup language format data items.

33. The web-based data server system of claim 32, wherein the mapping function is adapted to map SQL fields and records to XML tags.

34. The web-based data server system of claim 28, wherein the system is written in an open scripting language, the open scripting language permitting publication of scripts.

35. The web-based data server system of claim 28, wherein the system is adapted to be continuously running.

36. The web-based data server system of claim 28, further comprising a web-based administrator module, the administrator module for configuring and maintaining the web-based data server system, the administrator module comprising one or more web-based tools.

37. The web-based data server system of claim 36, wherein the administrator module comprises a tool adapted to allow mapping of structured data storage format items to markup language format items.

38. The web-based data server system of claim 36, wherein the markup language format items are XML tags and the structured data storage format items are database items in a relational database.

39. The web-based data server system of claim 36, wherein the markup language format items are HTML tags and the structured data storage format items are database items in a relational database.

40. The web-based data server system of claim 28, further comprising a web-enabled client, the client for converting markup language format data into structured data format data and storing the structured data format data into a data storage system.

41. The web-based data server system of claim 40, wherein the web-based client system is adapted to store data using a first data model, and the web based data server system is adapted to store data in a second data model, different from the first data model.