Methods for analyzing web sites using web services and devices thereof

• US 10,452,730 B2
• Filed: 12/22/2015
• Issued: 10/22/2019
• Est. Priority Date: 12/22/2015
• Status: Active Grant

• Current Assignee: Usablenet Incorporated
• Original Assignee: Usablenet Incorporated
• Inventors: Scoda, Enrico, Brondani, Marco
• Primary Examiner(s): Morrison, Jay A
• Assistant Examiner(s): Hoang, Ken
• Application Number:
  US14/977,995
• Publication Number:
  US 20170177727A1
• Time in Patent Office: 1,400 Days
• Field of Search: 709223, 709213, 709245, 709201, 709203, 709212, 709214, 709217, 709220, 709221, 709224, 709226, 709229, 709230, 709246, 719313, 719330, 719316, 719327, 719328, 717125, 717124, 717128, 717100, 717131, 717136, 717162, 717170
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links

Abstract

Methods, non-transitory computer readable media, and devices that determine when a job extracted from a stack is a pending job. When the determining indicates that the job is a pending job, a web service indicated in the job is executed. Another job is extracted from a web service response, and the another job is inserted into the stack. The web service is configured to obtain a web page to be analyzed based on the URL, execute the web page in an emulated JavaScript environment, and return the web service response. When the determining indicates that the job is not a pending job, a data collector event handler indicated in the job is executed. The data collector event handler is configured to update an output resource based on content of the analyzed web page included in the job.

Claims

1. A method for optimized analysis of structure or content of a web site, including one or more particular web pages included therein, using web services and an emulated JavaScript environment, the method implemented by one or more web site analyzer computing devices and comprising:

determining when a job extracted from a stack has a pending type and, when the determination indicates that the job has a pending type;

passing a Uniform Resource Locator (URL) included in the job as a parameter to a web service indicated in the job;

obtaining, by the web service indicated in the job, from a web server device, and based on the URL included in the job, a web page, executing, by the web service, the web page in an emulated JavaScript environment, and returning, by the web service, a web service response; and

extracting another job from the web service response and inserting the another job into the stack; and

when the determination indicates that the job does not have a pending type;

passing the job as a parameter to a data collector event handler indicated in the job, wherein the data collector event handler comprises a JavaScript function configured to update an output resource based on content of the analyzed web page included in the job extracted from the stack.

2. The method as set forth in claim 1, further comprising, when the determination indicates that the job has a pending type:

determining when the job is a duplicate job based on a canonical URL included in the job;

storing the canonical URL, when the determination indicates that the job is not a duplicate job; and

extracting an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determination indicates that the job is a duplicate job.

3. The method as set forth in claim 1, further comprising, when the determination indicates that the job has a pending type, inserting an end job having an end type into the stack before the another job and inserting a start job having a start type into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

4. The method as set forth in claim 1, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description of the web page, or a hash generated from the web page, and the job comprises a structured record and has:

a pending type and comprises one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start type and comprises one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end type and comprises one or more of the indication of the data collector event handler or the another name.

5. The method as set forth in claim 1, further comprising:

obtaining a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generating the job to have a pending type, an indication of the web service, and the URL; and

inserting the job into the stack.

6. The method as set forth in claim 1, further comprising:

determining when the stack is empty; and

providing the output resource when the determination indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.

7. A non-transitory computer readable medium having stored thereon programmed instructions for optimized analysis of structure or content of a web site, including one or more particular web pages included therein, using web services and an emulated JavaScript environment comprising executable code that, when executed by at least one processor, causes the processor to:

determine when a job extracted from a stack has a pending type and, when the determination indicates that the job has a pending type;

pass a Uniform Resource Locator (URL) included in the job as a parameter to a web service indicated in the job;

obtain, by the web service indicated in the job, from a web server device, and based on the URL included in the job, a web page, execute, by the web service, the web page in an emulated JavaScript environment, and return, by the web service, a web service response; and

extract another job from the web service response and insert the another job into the stack; and

when the determination indicates that the job does not have a pending type;

pass the job as a parameter to a data collector event handler indicated in the job, wherein the data collector event handler comprises a JavaScript function configured to update an output resource based on content of the analyzed web page included in the job extracted from the stack.

8. The non-transitory computer readable medium as set forth in claim 7, wherein the executable code, when executed by the processor, further causes the processor to, when the determination indicates that the job has a pending type:

determine when the job is a duplicate job based on a canonical URL included in the job;

store the canonical URL, when the determination indicates that the job is not a duplicate job; and

extract an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determination indicates that the job is a duplicate job.

9. The non-transitory computer readable medium as set forth in claim 7, wherein the executable code, when executed by the processor, further causes the processor to, when the determination indicates that the job has a pending type, insert an end job having an end type into the stack before the another job and inserting a start job having a start type into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

10. The non-transitory computer readable medium as set forth in claim 7, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description of the web page, or a hash generated from the web page, and the job comprises a structured record and has:

a pending type and comprises one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start type and comprises one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end type and comprises one or more of the indication of the data collector event handler or the another name.

11. The non-transitory computer readable medium as set forth in claim 7, wherein the executable code, when executed by the processor, further causes the processor to:

obtain a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generate the job to have a pending type, an indication of the web service, and the URL; and

insert the job into the stack.

12. The non-transitory computer readable medium as set forth in claim 7, wherein the executable code, when executed by the processor, further causes the processor to:

determine when the stack is empty; and

provide the output resource when the determination indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.

13. A web site analyzer computing device, comprising memory comprising programmed instructions stored thereon and one or more processors coupled to the memory and configured to execute the stored programmed instructions to:

determine when a job extracted from a stack has a pending type and, when the determination indicates that the job has a pending type;

pass a Uniform Resource Locator (URL) included in the job as a parameter to a web service indicated in the job;

obtain, by the web service indicated in the job, from a web server device, and based on the URL included in the job, a web page, execute, by the web service, the web page in an emulated JavaScript environment, and return, by the web service, a web service response; and

extract another job from the web service response and insert the another job into the stack; and

when the determination indicates that the job does not have a pending type;

pass the job as a parameter to a data collector event handler indicated in the job, wherein the data collector event handler comprises a JavaScript function configured to update an output resource based on content of the analyzed web page included in the job extracted from the stack.

14. The web site analyzer computing device as set forth in claim 13, wherein the processors are further configured to execute the stored programmed instructions to, when the determination indicates that the job has a pending type:

determine when the job is a duplicate job based on a canonical URL included in the job;

store the canonical URL, when the determination indicates that the job is not a duplicate job; and

extract an additional job from the stack, without executing the web service indicated in the job, extracting the another job from the web service response, or inserting the another job into the stack, when the determination indicates that the job is a duplicate job.

15. The web site analyzer computing device as set forth in claim 13, wherein the processors are further configured to execute the stored programmed instructions to, when the determination indicates that the job has a pending type, insert an end job having an end type into the stack before the another job and inserting a start job having a start type into the stack after the another job, wherein the start and end jobs identify another data collector event handler indicated in the web service response.

16. The web site analyzer computing device as set forth in claim 13, wherein the data collector event handler is a JavaScript function, the web service response comprises one or more of the another job, an indication of another data collector event handler, a description of the web page, or a hash generated from the web page, and the job comprises a structured record and has:

a pending type and comprises one or more of the indication of the web service, an indication of an endpoint of the web service, the URL of the web page to be analyzed, a name, or a canonical URL;

a start type and comprises one or more of an indication of the data collector event handler, another name, or the content of the analyzed web page obtained based on the execution of the web page in the emulated JavaScript environment;

or

an end type and comprises one or more of the indication of the data collector event handler or the another name.

17. The web site analyzer computing device as set forth in claim 13, wherein the processors are further configured to execute the stored programmed instructions to:

obtain a startup configuration file comprising one or more of an indication of the web service, the URL, a maximum number of web pages to process, or a function callback configured to validate the output resource;

generate the job to have a pending type, an indication of the web service, and the URL; and

insert the job into the stack.

18. The web site analyzer computing device as set forth in claim 13, wherein the processors are further configured to execute the stored programmed instructions to:

determine when the stack is empty; and

provide the output resource when the determination indicates that the stack is empty, wherein the output resource comprises an eXtensible Markup Language (XML) document, a JavaScript Object Notation (JSON) file, one or more records configured to be stored in a database, or an e-mail or other electronic communication.