RECYCLABLE PRIVATE MEMORY HEAPS FOR DYNAMIC SEARCH INDEXES

• US 20170061012A1
• Filed: 08/25/2015
• Published: 03/02/2017
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Sharma, Pranav, Bortnikov, Edward, Gueta, Guy
• Granted Patent: US 11,030,262 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0253 Garbage collection, i.e. reclamation of unreferenced memory
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/1028 Power efficiency
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/163 Server or database system
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

In one embodiment, a search engine may generate and store a plurality of search index segments such that each of the search index segments is stored in a corresponding one of a plurality of heaps of memory. The plurality of search index segments may include inverted index segments mapping content to documents containing the content. A garbage collection module may release one or more heaps of the memory.

Claims

1. A method, comprising:

generating and storing, by a search engine, a plurality of search index segments such that each of the plurality of search index segments is stored in a corresponding one of a plurality of heaps of memory, the plurality of search index segments including inverted index segments mapping content to documents containing the content; and

releasing, by a garbage collection module, one or more of the plurality of heaps of the memory.

2. The method of claim 1, further comprising:

storing, by the search engine, per-query objects in a heap of the memory that is separate from the plurality of heaps in which the plurality of search index segments are stored.

3. The method of claim 1, wherein the plurality of search index segments correspond to a plurality of generations of documents that are processed by the search engine.

4. The method of claim 1, wherein the memory is random access memory (RAM).

5. The method of claim 1, wherein the plurality of search index segments are generated and stored in real-time as documents are obtained, received, or processed.

6. The method of claim 1, further comprising:

for each one of the plurality of memory heaps, associating each one of two or more sections of the heap with a corresponding one of two or more central processing unit (CPU) cores.

7. The method of claim 1, further comprising:

storing, by a central processing unit (CPU) core, an object in one of two or more pools associated with the CPU core according to a size of the object;

wherein each one of the two or more pools is associated with a corresponding one of two or more ranges of object sizes.

8. The method of claim 1, further comprising:

discarding one or more of the plurality of search index segments; and

releasing one or more of the plurality of heaps of memory corresponding to the discarded search index segments.

15. The method of claim 1, wherein a youngest one of the plurality of search index segments has a read/write status associated therewith, and wherein remaining ones of the plurality of search index segments have a read status associated therewith.

20. The method of claim 1, further comprising:

storing, by a central processing unit (CPU) core, an object in one of two or more pools according to a size of the object;

wherein each one of the two or more pools is associated with a corresponding one of two or more ranges of object sizes.

9. A system, comprising:

one or more processors; and

a memory, at least one of the processors or memory being configured to;

generate and store, by a search engine, a plurality of search index segments such that each of the plurality of search index segments is stored in a corresponding one of a plurality of heaps of memory, the plurality of search index segments including inverted index segments mapping content to documents containing the content; and

release, by a garbage collection module, one or more of the plurality of heaps of the memory.

10. The system of claim 9, at least one of the processors or memory being further configured to:

store, by the search engine, per-query objects in a heap of the memory that is separate from the plurality of heaps in which the plurality of search index segments are stored.

11. The system of claim 9, wherein the plurality of search index segments correspond to a plurality of generations of documents that are processed by the search engine.

12. The system of claim 9, wherein the plurality of search index segments are generated and stored in real-time as documents are obtained, received, or processed.

13. The system of claim 9, wherein each one of the plurality of memory heaps comprises two or more sub-heaps, each of the two or more sub-heaps being associated with a corresponding one of two or more central processing unit (CPU) cores.

14. The system of claim 9, wherein each one of the plurality of memory heaps comprises two or more sub-heaps, wherein each one of the two or more sub-heaps is associated with a corresponding one of two or more ranges of object sizes.

16. At least one non-transitory computer-readable storage medium storing thereon computer-readable storage medium, comprising:

instructions configured to generate and store a plurality of search index segments such that each of the plurality of search index segments is stored in a corresponding one of a plurality of heaps of memory, the plurality of search index segments including inverted index segments mapping content to documents containing the content; and

instructions configured to release one or more of the plurality of heaps of the memory.

17. The non-transitory computer-readable storage medium of claim 16, further comprising:

instructions for storing per-query objects in a heap of the memory that is separate from the plurality of heaps in which the plurality of search index segments are stored.

18. The non-transitory computer-readable storage medium of claim 16, wherein a single one of the plurality of search index segments has read/write status.

19. The non-transitory computer-readable storage medium of claim 16, wherein each one of the plurality of memory heaps comprises two or more sub-heaps, each of the two or more sub-heaps being associated with a corresponding one of two or more central processing unit (CPU) cores.