System and method for automated discovery, binding, and integration of non-registered geospatial web services

• US 7,685,133 B2
• Filed: 05/24/2007
• Issued: 03/23/2010
• Est. Priority Date: 05/24/2006
• Status: Active Grant

• Current Assignee: The Government Of The United States As Represented By The Secretary Of The Navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Sample, John T, Shulman, Lev I, McCreedy, Frank P
• Primary Examiner(s): Wong; Leslie
• Application Number:
  US11/753,389
• Publication Number:
  US 20070276816A1
• Time in Patent Office: 1,034 Days
• Field of Search: 707/3, 707/10, 707/104.1, 707/103.R, 707/103.Y, 707/E17.061, 707/E17.11, 707/E17.121, 709/203, 709/219
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: Y10S 707/919 Geographic
• CPC Class Codes: Y10S 707/921 Spatial

Abstract

A method and computer system for identifying internet web pages containing documents that comply with a predetermined XML schema. The method includes searching the internet with a search engine for web pages using initial search terms and identifying a first set of HTTP URLs, web crawling at least the first set of HTTP URLs to identify additional HTTP URLs, appending a query to the identified URLs, and evaluating the responses to the query to determine which responses comply with the predetermined XML schema. The XML schema can be a Web Mapping Services schema. The system can store responses that comply with the XML schema in a database of servers, periodically check the database for validity, and convert the map requests for map servers in the database to a GIDB Portal Interface API.

Claims

1. A computer-based method for identifying internet web pages containing documents that comply with a predetermined XML schema, comprising:

searching the internet with a search engine for web pages using initial search terms and identifying a first set of HTTP URLs;

web crawling at least the first set of HTTP URLs to identify additional HTTP URLs;

appending a query to the identified URLs; and

evaluating the responses to the query to determine which responses comply with the predetermined XML schema; and

storing the HTTP URLs responses that comply with the predetermined XML schema in a database.

2. The method according to claim 1, further comprising:

the web crawler selecting URLs of the first set of HTTP URLs that are a document type that is unlikely to contain more HTTP URLs, and not web crawling the selected URLs.

3. The method according to claim 2, wherein the selected URLs are at least one of music files, image files, and executable files.

4. The method according to claim 1, further comprising:

comparing a newly discovered document against previously known WMS server documents to avoid duplication.

5. The method according to claim 1, further comprising:

adding the HTTP URLs responses that comply with the predetermined XML schema to a database of available servers.

6. The method according to claim 4, further comprising web crawling the HTTP URLS in the database periodically to discover non-operational links.

7. The method according to claim 1, wherein the web crawling includes web crawling a predetermined set of additional web sites.

8. The method according to claim 1, wherein predetermined set of additional web sites include GIS web sites.

9. The method according to claim 1, wherein the predetermined XML schema is a schema published by the Open Geospatial Consortium.

10. The method according to claim 1, wherein the appended query requests a Web Mapping Services Capabilities document.

11. The method according to claim 1, wherein the appended query includes ?REQUEST=GetCapabilities.

12. The method according to claim 1, wherein the appended query includes ?REQUEST=GetCapabilities&

SERVICE=WMS.

13. The method according to claim 1, wherein said searching includes searching using an API.

14. The method according to claim 1, further comprising:

repeating said searching, said web crawling, said appending and said evaluating at periodic intervals.

15. The method according to claim 5, further comprising:

converting map requests for at least one particular map server in the database to a GIDB Portal Interface API.

16. A computer system including a server connected to the internet and having software for identifying internet web pages containing documents that comply with a predetermined XML schema, the software configured to:

search the internet with a search engine for web pages using initial search terms and identify a first set of HTTP URLs;

web crawl at least the first set of HTTP URLs to identify additional HTTP URLs;

append a query to the identified URLs;

evaluate the responses to the query to determine which responses comply with the predetermined XML schema; and

store a list of valued URLs having responses with the predetermined XML schema.

17. The system according to claim 16, further comprising:

the web crawler selecting URLs of the first set of HTTP URLs that are a document type that is unlikely to contain more HTTP URLs, and not web crawling the selected URLs.

18. The system according to claim 17, wherein the selected URLs are at least one of music files, image files, and executable files.

19. The system according to claim 16, further comprising:

comparing a newly discovered document against previously known WMS server documents to avoid duplication.

20. The system according to claim 16, further comprising:

adding the HTTP URLs responses that comply with the predetermined XML schema to a database of available servers.

21. The system according to claim 19, further comprising web crawling the HTTP URLS in the database periodically to discover non-operational links.

22. The system according to claim 16, wherein the web crawling includes web crawling a predetermined set of additional web sites.

23. The system according to claim 17, wherein predetermined set of additional web sites include GIS web sites.

24. The system according to claim 17, wherein the predetermined XML schema is a schema published by the Open Geospatial Consortium.

25. The system according to claim 17, wherein the appended query requests a Web Mapping Services Capabilities document.

26. The system according to claim 16, wherein the appended query includes ?REQUEST=GetCapabilities.

27. The system according to claim 16, wherein the appended query includes ?REQUEST=GetCapabilities&

SERVICE=WMS.

28. The system according to claim 16, wherein said searching includes searching using an API.

29. The system according to claim 16, further comprising:

repeating said searching, said web crawling, said appending and said evaluating at periodic intervals.

30. The system according to claim 16, further comprising:

converting map requests for at least one particular map server in the database to a GIDB Portal Interface API.