INFRASTRUCTURE ENABLING INTELLIGENT EXECUTION AND CRAWLING OF A WEB APPLICATION

• US 20160335353A1
• Filed: 07/26/2016
• Published: 11/17/2016
• Est. Priority Date: 04/06/2010
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Keyani, Pedram, Gianos, Nicholas Philip, Jeffries, Paul Christopher, Gowda, Chetan, Kar, Siddharth
• Granted Patent: US 10,642,904 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines

Abstract

In particular embodiments, a method comprises accessing, by one or more computing systems associated with a social-networking system, a structured document of a network application, the structured document comprising structural information and content comprising one or more embedded scripts, resources, or identifiers for the resources. The method further comprises processing the structured document to generate a model representation of the structured document, executing at least some of the content of the structured document and logging multiple snapshots of the model representation of the structured document as the model representation is generated in response to one or more interactions initiated by execution of the content. The method further comprises creating a behavior model of the network application based on the multiple snapshots of the model representation of the structured document and determining, based on the behavior model, compliance by the network application with one or more requirements of the social-networking system.

Claims

1. A method comprising:

accessing, by one or more computing systems associated with a social-networking system, a structured document of a network application, the structured document comprising structural information and content comprising one or more embedded scripts, resources, or identifiers for the resources;

processing, by the one or more computing systems, the structured document to generate a model representation of the structured document;

executing, by the one or more computing systems, at least some of the content of the structured document;

logging, by the one or more computing systems, multiple snapshots of the model representation of the structured document as the model representation is generated in response to one or more interactions initiated by the executing of the content;

creating, by the one or more computing systems, a behavior model of the network application based on the multiple snapshots of the model representation of the structured document; and

determining, by the one or more computing devices, based on the behavior model, compliance by the network application with one or more requirements of the social-networking system.

2. The method of claim 1, further comprising:

enumerating one or more attributes of the structured document, and wherein one or more of the enumerated attributes are included in the behavior model.

3. The method of claim 2, wherein the determining compliance comprises comparing the one or more attributes to one or more profiles to identify an undesirable application.

4. The method of claim 3, wherein the one or more attributes are selected based on a scripted rule set.

5. The method of claim 1, wherein the behavior model further comprises one or more URLs or domain names of one or more resources for which requests were sent to or received from.

6. The method of claim 1, wherein the behavior model further comprises one or more URLs or domain names corresponding to one or more advertisement developers or advertisement provider networks to which requests for advertisements were transmitted by the web application or from which one or more incoming responses including advertisements were received.

7. The method of claim 1, further comprising:

generating a log that comprises URLs or domain names corresponding to ad networks ascertained after filtering the behavior model; and

querying the log against a list of known rogue ad networks.

8. The method of claim 7, further comprising:

capturing one or more parameters sent to one or more computers associated with the URLs or domain names; and

determining whether the network application is passing data received from the social-networking system to other parties.

9. The method of claim 1, further comprising:

determining, based on the behavior model, how the network application appears at one or more points in a particular user flow.

10. The method of claim 1, further comprising:

recording one or more variations in the behavior model of the network application based on characteristics of a logged-in user of the social-networking system.

11. The method of claim 10, wherein the characteristics comprise an identity of the logged-in user, demographics of the logged-in user, or profile information from the logged-in user'"'"'s profile in the social-networking system.

12. The method of claim 1, further comprising:

recording one or more variations in the behavior model of the network application based on a geographic location, a browser type, or a type of computing device of the one or more computing systems.

13. The method of claim 1, wherein:

the one or more computing systems include a primary computing system and one or more secondary computing systems;

each of the one or more secondary computing systems hosts one or more crawler processes each operable to access and render the network application; and

wherein the method further comprises accessing, by the one or more computing systems, the structured document comprises receiving, by one of the crawler processes executing within one of the secondary computing systems, a request from the primary computer system to access the network application.

14. The method of claim 13, further comprising accessing, by the one of the crawler processes, one or more servers hosting a canvas web page.

15. The method of claim 14, further comprising logging into, by the one of the crawler processes, the one or more servers using test user credentials.

16. The method of claim 13, wherein each crawler process is implemented, at least in part, with all or portions of a cross platform component model and a layout engine.

17. The method of claim 16, wherein:

each crawler process further comprises an overlying programming layer overtop of the cross platform component model and layout engine layers, and

the logging multiple snapshots of the model representation of the structured document further comprises tracking the one or more interactions initiated by the executing of the content performed by the overlying programming layer.

18. The method of claim 1, wherein the model representation is a Document Object Model (DOM) representation.

19. A system comprising:

one or more processors associated with a social-networking system; and

logic encoded in one or more computer-readable tangible storage media that, when executed by the one or more processors, is operable to;

access a structured document of a network application, the structured document comprising structural information and content comprising one or more embedded scripts, resources, or identifiers for the resources;

process the structured document to generate a model representation of the structured document;

execute at least some of the content of the structured document;

log multiple snapshots of the model representation of the structured document as the model representation is generated in response to one or more interactions initiated by the executing of the content;

create a behavior model of the network application based on the multiple snapshots of the model representation of the structured document; and

determine, based on the behavior model, compliance by the network application with one or more requirements of the social-networking system.

20. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a structured document of a network application, the structured document comprising structural information and content comprising one or more embedded scripts, resources, or identifiers for the resources;

process the structured document to generate a model representation of the structured document;

execute at least some of the content of the structured document;

log multiple snapshots of the model representation of the structured document as the model representation is generated in response to one or more interactions initiated by the executing of the content;

create a behavior model of the network application based on the multiple snapshots of the model representation of the structured document; and

determine, based on the behavior model, compliance by the network application with one or more requirements of the social-networking system.