METHODS FOR ANALYZING WEB SITES USING WEB SERVICES AND DEVICES THEREOF

• US 20170177727A1
• Filed: 12/22/2015
• Published: 06/22/2017
• Est. Priority Date: 12/22/2015
• Status: Active Grant

• Current Assignee: Usablenet Incorporated
• Original Assignee: Usablenet Incorporated
• Inventors: Scoda, Enrico, Brondani, Marco
• Granted Patent: US 10,452,730 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links

Abstract

Methods, non-transitory computer readable media, and devices that determine when a job extracted from a stack is a pending job. When the determining indicates that the job is a pending job, a web service indicated in the job is executed. Another job is extracted from a web service response, and the another job is inserted into the stack. The web service is configured to obtain a web page to be analyzed based on the URL, execute the web page in an emulated JavaScript environment, and return the web service response. When the determining indicates that the job is not a pending job, a data collector event handler indicated in the job is executed. The data collector event handler is configured to update an output resource based on content of the analyzed web page included in the job.

Claims

1. A method for analyzing web sites using web services, the method comprising:

determining, by the web site analyzer computing device, when a job extracted from a stack is a pending job;

executing, by the web site analyzer computing device, a web service indicated in the job comprising passing a Uniform Resource Locator (URL) included in the job as a parameter to the web service, extracting another job from a web service response, and inserting the another job into the stack, when the determining indicates that the job is a pending job, wherein the web service is configured to obtain a web page to be analyzed based on the URL, execute the web page in an emulated JavaScript environment, and return the web service response; and

executing, by the web site analyzer computing device, a data collector event handler indicated in the job comprising passing the job as a parameter to the data collector event handler, when the determining indicates that the job is not a pending job, wherein the data collector event handler is configured to update an output resource based on content of the analyzed web page included in the job.

2. The method as set forth in claim 1, further comprising, when the determining indicates that the job is a pending job:

determining, by the web site analyzer computing device, when the job is a duplicate job based on a canonical URL included in the job;

storing, by the web site analyzer computing device, the canonical URL, when the determining indicates that the job is not a duplicate job; and

extracting, by the web site analyzer computing device, an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determining indicates that the job is a duplicate job.

3. The method as set forth in claim 1, further comprising, when the determining indicates that the job is a pending job, inserting, by the web site analyzer computing device, an end job into the stack before the another job and inserting a start job into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

4. The method as set forth in claim 1, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description the web page, or a hash generated from the web page, and the job comprises a structured record and is:

a pending job comprising one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start job comprising one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end job comprising one or more of the indication of the data collector event handler or the another name.

5. The method as set forth in claim 1, further comprising:

obtaining, by the web site analyzer computing device, a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generating, by the web site analyzer computing device, the job to have a pending type, an indication of the web service, and the URL; and

inserting, by the web site analyzer computing device, the job into the stack.

6. The method as set forth in claim 1, further comprising:

determining, by the web site analyzer computing device, when the stack is empty; and

providing, by the web site analyzer computing device, the output resource when the determining indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.

7. A non-transitory computer readable medium having stored thereon programmed instructions for analyzing web sites using web services comprising executable code that, when executed by at least one processor, causes the processor to perform steps comprising:

determining when a job extracted from a stack is a pending job;

executing a web service indicated in the job comprising passing a Uniform Resource Locator (URL) included in the job as a parameter to the web service, extracting another job from a web service response, and inserting the another job into the stack, when the determining indicates that the job is a pending job, wherein the web service is configured to obtain a web page to be analyzed based on the URL, execute the web page in an emulated JavaScript environment, and return the web service response; and

executing a data collector event handler indicated in the job comprising passing the job as a parameter to the data collector event handler, when the determining indicates that the job is not a pending job, wherein the data collector event handler is configured to update an output resource based on content of the analyzed web page included in the job.

8. The non-transitory computer readable medium as set forth in claim 7, further having stored thereon one or more additional programmed instructions comprising executable code that, when executed by the processor further cause the processor to perform one or more additional steps comprising, when the determining indicates that the job is a pending job:

determining when the job is a duplicate job based on a canonical URL included in the job;

storing the canonical URL, when the determining indicates that the job is not a duplicate job; and

extracting an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determining indicates that the job is a duplicate job.

9. The non-transitory computer readable medium as set forth in claim 7, further having stored thereon one or more additional programmed instructions comprising executable code that, when executed by the processor further cause the processor to perform one or more additional steps comprising, when the determining indicates that the job is a pending job, inserting an end job into the stack before the another job and inserting a start job into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

10. The non-transitory computer readable medium as set forth in claim 7, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description the web page, or a hash generated from the web page, and the job comprises a structured record and is:

a pending job comprising one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start job comprising one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end job comprising one or more of the indication of the data collector event handler or the another name.

11. The non-transitory computer readable medium as set forth in claim 7, further having stored thereon one or more additional programmed instructions comprising executable code that, when executed by the processor further cause the processor to perform one or more additional steps comprising:

obtaining a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generating the job to have a pending type, an indication of the web service, and the URL; and

inserting the job into the stack.

12. The non-transitory computer readable medium as set forth in claim 7, further having stored thereon one or more additional programmed instructions comprising executable code that, when executed by the processor further cause the processor to perform one or more additional steps comprising:

determining when the stack is empty; and

providing the output resource when the determining indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.

13. A web site analyzer computing device, comprising one or more processors coupled to a memory and configured to execute programmed instructions comprising and stored in the memory to:

determine when a job extracted from a stack is a pending job;

execute a web service indicated in the job and pass a Uniform Resource Locator (URL) included in the job as a parameter to the web service, extract another job from a web service response, and insert the another job into the stack, when the determining indicates that the job is a pending job, wherein the web service is configured to obtain a web page to be analyzed based on the URL, execute the web page in an emulated JavaScript environment, and return the web service response; and

execute a data collector event handler indicated in the job and pass the job as a parameter to the data collector event handler, when the determining indicates that the job is not a pending job, wherein the data collector event handler is configured to update an output resource based on content of the analyzed web page included in the job.

14. The web site analyzer computing device as set forth in claim 13, wherein the processor are further configured to execute one or more additional programmed instructions comprising and stored in the memory to, when the determining indicates that the job is a pending job:

determine when the job is a duplicate job based on a canonical URL included in the job;

store the canonical URL, when the determining indicates that the job is not a duplicate job; and

extract an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determining indicates that the job is a duplicate job.

15. The web site analyzer computing device as set forth in claim 13, wherein the processor are further configured to execute one or more additional programmed instructions comprising and stored in the memory to, when the determining indicates that the job is a pending job, insert an end job into the stack before the another job and insert a start job into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

16. The web site analyzer computing device as set forth in claim 13, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description the web page, or a hash generated from the web page, and the job comprises a structured record and is:

a pending job comprising one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start job comprising one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end job comprising one or more of the indication of the data collector event handler or the another name.

17. The web site analyzer computing device as set forth in claim 13, wherein the processor are further configured to execute one or more additional programmed instructions comprising and stored in the memory to:

obtain a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generate the job to have a pending type, an indication of the web service, and the URL; and

insert the job into the stack.

18. The web site analyzer computing device as set forth in claim 13, wherein the processor are further configured to execute one or more additional programmed instructions comprising and stored in the memory to:

determine when the stack is empty; and

provide the output resource when the determining indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.