THEMATIC WEB CORPUS

• US 20170140055A1
• Filed: 11/17/2016
• Published: 05/18/2017
• Est. Priority Date: 11/17/2015
• Status: Active Grant

• Current Assignee: Dassault Systemes SA
• Original Assignee: Dassault Systemes SA
• Inventors: GREHANT, Xavier, Champenois, Morgan
• Granted Patent: US 10,783,196 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The invention notably relates to a computer-implemented method, performed by a server storing an index of a search engine, for sending, to a client, the URLs of pages of a Web corpus that relates to a theme. The method comprises receiving, from the client, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword; determining in the index the group that consists of the URLs of all pages that match the query; and sending to the client the URLs of the group as a stream.

Such a method improves the building of a thematic Web corpus.

Claims

1. Computer-implemented method for building a Web corpus that relates to a theme, the method comprising, by a server storing an index of a search engine, sending, to a client, the URLs of pages of a Web corpus that relates to the theme, including:

receiving, from the client, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

determining in the index the group that consists of the URLs of all pages that match the query, wherein the determining consists in;

reading the keywords of the disjunction of the query on the index, thereby retrieving at least one set of URLs from the index, then

performing on the retrieved at least one set of URLs a scheme of set operations that corresponds to the disjunction of the query, thereby

leading to the group of URLs; and

sending to the client the URLs of the group as a stream.

2. The method of claim 1 wherein the sending to the client of the URLs of the group as a stream comprises:

establishing a network connection with the client;

streaming the URLs of the group on the network connection;

then

closing the network connection.

3. The method of claim 2, wherein the network connection is a HTTP connection.

4. Computer-implemented method of claim 1, wherein the method further comprises, prior to receiving the structured query from the client, sending the structured query by the client to the server.

5. The method of claim 4, wherein the method further comprises, by the client, storing locally the URLs received from the server as a stream.

6. The method of claim 4, wherein the method then further comprises, by the client, crawling the pages of the URLs received from the server or transmitting the URLs received from the server to a Web crawler.

7. Computer-implemented method, performed by a client, for building a Web corpus that relates to a theme, wherein the method comprises:

sending, to a server, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

then

receiving from the server the URLs of pages of the Web corpus as a stream.

8. The method of claim 7, wherein the method further comprises storing locally the URLs received from the server as a stream.

9. The method of claim 7, wherein the method then further comprises crawling the pages of the URLs received from the server or transmitting the URLs received from the server to a Web crawler.

10. A non-transitory computer-readable medium having recorded thereon a computer program comprising instructions for performing a computer-implemented method for building a Web corpus that relates to a theme, the method comprising, by a server storing an index of a search engine, sending, to a client, the URLs of pages of a Web corpus that relates to the theme, including:

receiving, from the client, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

determining in the index the group that consists of the URLs of all pages that match the query, wherein the determining consists in;

reading the keywords of the disjunction of the query on the index, thereby

retrieving at least one set of URLs from the index, then

performing on the retrieved at least one set of URLs a scheme of set operations that corresponds to the disjunction of the query, thereby

leading to the group of URLs; and

sending to the client the URLs of the group as a stream.

11. A non-transitory computer-readable medium having recorded thereon a computer program comprising instructions for performing a computer-implemented method for building a Web corpus that relates to a theme, the method comprising, by a client:

sending, to a server, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

then

receiving from the server the URLs of pages of the Web corpus as a stream.

12. A server system comprising a processor coupled to a memory having recorded thereon a search engine index and a computer program for performing a computer-implemented method for building a Web corpus that relates to a theme, the method comprising sending, to a client, the URLs of pages of a Web corpus that relates to the theme, including:

receiving, from the client, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

determining in the index the group that consists of the URLs of all pages that match the query, wherein the determining consists in;

reading the keywords of the disjunction of the query on the index, thereby

retrieving at least one set of URLs from the index, then

performing on the retrieved at least one set of URLs a scheme of set operations that corresponds to the disjunction of the query, thereby

leading to the group of URLs; and

sending to the client the URLs of the group as a stream.

13. The server system of claim 12, wherein the sending to the client of the URLs of the group as a stream comprises:

establishing a network connection with the client;

streaming the URLs of the group on the network connection;

then

closing the network connection.

14. The server system of claim 13, wherein the network connection is a HTTP connection.

15. A client system comprising a processor coupled to a memory having recorded thereon a computer program comprising instructions for performing a computer-implemented method for building a Web corpus that relates to a theme, the method comprising:

sending, to a server, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword;

then

receiving from the server the URLs of pages of the Web corpus as a stream.

16. The method of claim 15, wherein the method further comprises storing locally the URLs received from the server as a stream.

17. The method of claim 15, wherein the method then further comprises crawling the pages of the URLs received from the server or transmitting the URLs received from the server to a Web crawler.