Indexing of URLs with fragments

• US 8,468,145 B2
• Filed: 11/10/2011
• Issued: 06/18/2013
• Est. Priority Date: 09/16/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fedorynski, Pawel Aleksander, Kapoor, Rupesh
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US13/293,984
• Publication Number:
  US 20130073536A1
• Time in Patent Office: 586 Days
• Field of Search: 707/736, 707/722, 707/768, 707/728, 707/706, 707/709, 707/741, 707/811, 707/E17.002, 707/E17.108, 709/246, 709/231, 709/204, 709/203, 709/218, 709/219, 715/234, 715/239, 715/760, 715/230, 715/231
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links

Abstract

A URL inspector may determine a uniform resource locator (URL) which includes an indexable fragment. A URL separator may separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment. An indexer may process content of the base URL to obtain processed content thereof. A rendering system may render the processed content together with the URL which includes the indexable fragment to obtain rendered content. A content converter may convert the rendered content into indexable content.

Claims

1. A computer system including instructions stored on a non-transitory computer-readable storage device and executable by at least one processor, the computer system comprising:

a URL inspector configured to cause the at least one processor to determine a uniform resource locator (URL) which includes an indexable fragment, the indexable fragment being associated with dynamic content rendered in a website identified by the URL which includes the indexable fragment, the dynamic content being associated with executable code configured to be executed by a browser rendering the website to thereby perform a partial modification of a previously-rendered version of the website;

a URL separator configured to cause the at least one processor to separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment;

an indexer configured to cause the at least one processor to process content identified by the base URL to obtain processed content thereof, wherein the processed content does not include embedded resources to be rendered;

a rendering system configured to cause the at least one processor to render the processed content together with content identified by the URL which includes the indexable fragment to obtain rendered content, wherein the rendered content includes rendering of the embedded resources; and

a content converter configured to cause the at least one processor to convert the rendered content into indexable content, the indexable content including markup language code.

2. The system of claim 1 wherein the URL inspector is configured to determine the URL which includes the indexable fragment in conjunction with a crawling of a website which includes a link to the URL which includes the indexable fragment.

3. The system of claim 1 wherein the URL inspector is configured to distinguish the indexable fragment from a pointer fragment designed to identify a specific portion of a website.

4. The system of claim 1 wherein the URL separator is configured to store the base URL in association with the URL which includes the indexable fragment, using one or more memories.

5. The system of claim 4, wherein the association includes a common identifier assigned to the base URL and to the URL which includes the indexable fragment within the at least one memory.

6. The system of claim 1 wherein the indexer is configured to process the content of the base URL including crawling the content of the base URL.

7. The system of claim 1 wherein the indexer is configured to process the base URL to obtain the processed content in association with a subsequent re-crawling of the content of the base URL.

8. The system of claim 1 wherein the rendering system is configured to render the processed content to obtain the rendered content, including executing included executable code designed to be executed by a client-side application.

9. The system of claim 1 wherein the rendering system is configured to render the processed content to obtain the rendered content, including outputting a document object model (DOM) tree within the rendered content, the DOM tree including dynamically updated portions of the website associated with the indexable fragment.

10. The system of claim 9 wherein the content converter is configured to convert the rendered content including serializing the DOM tree to obtain HyperText Markup Language (HTML) code.

11. The system of claim 1, wherein the indexable fragment is a fragment included within the URL and providing identification of a client side execution and a dynamic updating of an underlying website.

12. A computer-implemented method including executing instructions stored on a non-transitory computer-readable storage device, the method comprising:

determining a uniform resource locater (URL) which includes an indexable fragment, the indexable fragment being associated with dynamic content rendered in a website identified by the URL which includes the indexable fragment, the dynamic content being associated with executable code configured to be executed by a browser rendering the website to thereby perform a partial modification of a previously-rendered version of the website;

separating, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment;

processing content identified by the base URL to obtain processed content thereof, wherein the processed content does not include embedded resources to be rendered;

rendering the processed content together with content identified by the URL which includes the indexable fragment to obtain rendered content, wherein the rendered content includes rendering of the embedded resources; and

converting the rendered content into indexable content, the indexable content including markup language code.

13. The method of claim 12, wherein determining the URL which includes the indexable fragment is performed in conjunction with a crawling of a website which includes a link to the URL which includes the indexable fragment.

14. A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage device and including executable code that, when executed, is configured to cause a data processing apparatus to:

determine a uniform resource locater (URL) which includes an indexable fragment, the indexable fragment being associated with dynamic content rendered in a website identified by the URL which includes the indexable fragment the dynamic content being associated with executable code configured to be executed by a browser rendering the website to thereby perform a partial modification of a previously-rendered version of the website;

separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment;

process content identified by the base URL to obtain processed content thereof, wherein the processed content does not include embedded resources to be rendered;

render the processed content together with content identified by the URL which includes the indexable fragment to obtain rendered content, wherein the rendered content includes rendering of the embedded resources; and

convert the rendered content into indexable content, the indexable content including markup language code.

15. The computer program product of claim 14, wherein the URL which includes the indexable fragment is determined in conjunction with a crawling of a website which includes a link to the URL which includes the indexable fragment.

16. The computer program product of claim 14, wherein the base URL is stored in association with the URL which includes the indexable fragment, using one or more memories.

17. The computer program product of claim 14, wherein the processed content is rendered to obtain the rendered content, including executing included executable code designed to be executed by a client-side application.

18. The computer program product of claim 14 wherein the processed content includes crawled content of the base URL.