BATCH-OPTIMIZED RENDER AND FETCH ARCHITECTURE

• US 20150379014A1
• Filed: 10/22/2014
• Published: 12/31/2015
• Est. Priority Date: 06/26/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Xu, Hui, Kapoor, Rupesh, Hendriks, Erik Arjan, Fang, Hao, Tapus, Cristian
• Granted Patent: US 9,984,130 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/186 Templates

Abstract

Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Claims

1. A computer system comprising:

at least one processor; and

memory storing;

a data store of content for embedded items, and

instructions that, when executed by the at least one processor, cause the system to;

receive a request, from a batch process, to render a web page,

identify an embedded item in the web page,

determine, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item,

in response to the determination, provide the content for the previously fetched embedded item from the data store,

generate a rendering result for the web page using the content for the previously fetched embedded item, and

provide the rendering result to the batch process.

2. The system of claim 1, wherein as part of determining that the embedded item has content that is duplicative of content for a previously fetched embedded item, the instructions further include instructions that, when executed by the at least one processor, cause the system to:

match the embedded item to a template of the rewrite rule, the rewrite rule also including a redirect identifier,

wherein providing the content for the previously fetched embedded item includes using the redirect identifier to locate the content for the previously fetched embedded item.

3. The system of claim 2, wherein the template includes a URL without a query string.

4. The system of claim 1, wherein the embedded item is a first embedded item and the instructions further include instructions that, when executed by the at least one processor, cause the system to:

identify a second embedded item in the web page;

determine whether the second embedded item is blacklisted;

return an error when the second embedded item is blacklisted without fetching content for the second embedded item; and

generate the rendering result without the content for the second embedded item.

5. The system of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

use a virtual clock when generating the rendering result,

wherein the virtual clock advances independently of real time.

6. The system of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

use a virtual clock when generating the rendering result,

wherein the virtual clock does not advance while waiting for the provided content of the previously fetched embedded item.

7. The system of claim 1, wherein the embedded item is a first embedded item and the instructions further include instructions that, when executed by the at least one processor, cause the system to:

identify a second embedded item in the web page;

determine that the second embedded item includes an image;

generate a mock image that specifies dimensions for the second embedded item using a dimension table; and

use the mock image in generating the rendering result.

8. A method comprising:

receiving a request, from a batch process, to render a web page;

initializing, using at least one processor, a virtual clock and a task list for rendering the web page, wherein the virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run;

generating, using the at least one processor, a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list; and

providing the rendering result to the batch process.

9. The method of claim 8, wherein initializing the task list includes adding the stop task with a run time set to a predetermined time added to the virtual clock.

10. The method of claim 8, wherein the batch processes includes an indexing engine and the method further comprises using the rendering result to rank tokens in an index.

11. The method of claim 8, further comprising advancing the virtual clock to a run time of a task in the task list when no requests for embedded items are outstanding and only tasks with run times greater than the virtual clock are in the task list.

12. The method of claim 8, further comprising:

identifying an embedded image in the web page;

requesting content for the embedded image;

receiving, in response to the request, a mock image that specifies dimensions for the embedded image but has empty content; and

using the mock image in generating the rendering result.

13. The method of claim 8, wherein the batch process is an indexing engine and the method further comprises demoting a rank for the web page based on information in the rendering result.

14. The method of claim 8, wherein the batch process is an indexing engine and the method further comprises using the rendering result to index dynamically generated content.

15. The method of claim 8, wherein the rendering result includes an image and a layout that identifies coordinates of elements in the web page in the image.

16. A method comprising:

receiving a request from a batch rendering process for Uniform Resource Locator (URL) of an embedded item in a web page;

applying, using at least one processor, rewrite rules to determine a rewritten URL;

determining, using the at least one processor, whether content for the rewritten URL exists in a data store;

when the content exists, providing the content to the batch rendering process; and

when the content does not exist;

initiating a fetch of the content, wherein the batch rendering process is configured to wait without timing out during the fetch,

receiving the content from a web-crawling engine,

providing the content to the batch rendering process, and

storing the content in the data store.

17. The method of claim 16, further comprising using, by the batch rendering process, the content to generate a rendering result of the web page.

18. The method of claim 17, wherein the rendering result includes layout information and dynamically-generated content.

19. The method of claim 16, wherein applying rewrite rules includes:

matching the URL to a template, the template being associated with a redirect URL,

wherein when the URL matches the template, the redirect URL is determined to be the rewritten URL, and

wherein when the URL fails to match a template, the URL is determined to be the rewritten URL.

20. The method of claim 16, the method further comprising:

determining that the content for the rewritten URL is stale based on a change rate or a type of the embedded item stored in the data store; and

in response to the determination that the content for the rewritten URL is stale;

receiving updated content from the web-crawling engine,

updating the data store with the updated content, and

providing the updated content as the content for rewritten URL.