Method and system for spilling from a queue to a persistent store

• US 8,145,859 B2
• Filed: 03/02/2009
• Issued: 03/27/2012
• Est. Priority Date: 03/02/2009
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Park, Hoyong, Jain, Namit, Srinivasan, Anand, Mishra, Shailendra
• Primary Examiner(s): SONG, HUA JASMINE
• Application Number:
  US12/396,008
• Publication Number:
  US 20100223437A1
• Time in Patent Office: 1,121 Days
• Field of Search: 711133-135, 711159-160, 711/E12.002
• US Class Current: 711/159
• CPC Class Codes: G06F 12/12 Replacement control

Abstract

Techniques for managing memory usage of a processing system by spilling data from a memory to a persistent store based upon an evict policy are provided. A triggering event is detected. In response to the triggering event and based on the evict policy, it is determined whether data from the memory of the processing system is to be spilled to the persistent storage. The determination is made by comparing a level of free memory of the processing system with a threshold specified by the evict policy. The data is evicted from the memory.

Claims

1. A method performed by a data stream processing system for managing memory usage of the data stream processing system, the method comprising:

detecting a triggering event;

responsive to the triggering event, determining, based upon an evict policy, whether data from a memory of the data stream processing system is to be spilled to a persistent storage, wherein the memory includes a plurality of buffer queues, and wherein the determining comprises;

determining a threshold from a plurality of thresholds associated with a current state of the data stream processing system based upon the evict policy; and

comparing the threshold with a level of free memory of the data stream processing system; and

evicting the data from the memory.

2. The method of claim 1, further comprising:

storing the data to the persistent storage.

3. The method of claim 1, wherein the triggering event is receiving an enqueue request or a dequeue request.

4. The method of claim 1, further comprising:

transitioning the data stream processing system from the current state to a new state based upon the comparison.

5. The method of claim 1, wherein the current state of the data stream processing system is associated with one or more thresholds of the plurality of thresholds.

6. The method of claim 1, further comprising:

selecting a buffer queue of the plurality of buffer queues from which to perform eviction based upon the evict policy, wherein the data is evicted from the selected buffer queue.

7. The method of claim 1, wherein the evicted data comprises one or more pages of a data stream.

8. The method of claim 7, wherein the eviction of the one or more pages occurs from the largest buffer queue of the plurality of buffer queues.

9. The method of claim 7, wherein the eviction of the one or more pages occurs from each buffer queue of the plurality of buffer queues.

10. The method of claim 7, wherein the eviction of the one or more pages is executed by a thread independent from a caller thread.

11. The method of claim 1, wherein prior to eviction, a buffer queue of the plurality of buffer queues operates according to a memory mode where spilling is not performed.

12. A computer-readable storage medium storing a computer program product for managing memory usage of a data stream processing system, the computer program product comprising:

computer code for detecting a triggering event;

computer code for responsive to the triggering event, determining, based upon an evict policy, whether data from a memory of the data stream processing system is to be spilled to a persistent storage, wherein the memory includes a plurality of buffer queues, and wherein the determining comprises;

determining a threshold from a plurality of thresholds associated with a current state of the data stream processing system based upon the evict policy; and

comparing the threshold with a level of free memory of the data stream processing system; and

computer code for evicting the data from the memory.

13. The computer program product according to claim 12, wherein the triggering event is receiving an enqueue request or a dequeue request.

14. The computer program product according to claim 12, wherein the current state of the processing system is associated with one or more thresholds of the plurality of thresholds.

15. The computer program product according to claim 12, the computer program product further comprising:

computer code for selecting a buffer queue of the plurality of buffer queues from which to perform eviction based upon the evict policy, wherein the data is evicted from the selected buffer queue.

16. A system comprising:

a processor configured to;

detect a triggering event;

responsive to the triggering event, determine, based upon an evict policy, whether data from a memory of a data stream processing system is to be spilled to a persistent storage, wherein the memory includes a plurality of buffer queues, and wherein the determining comprises;

determining a threshold from a plurality of thresholds associated with a current state of the data stream processing system based upon the evict policy; and

comparing the threshold with a level of free memory of the data stream processing system; and

evict the data from the memory.

17. The system according to claim 16, wherein the triggering event is receiving an enqueue request or a dequeue request.

18. The system according to claim 16, wherein the current state of the processing system is associated with one or more thresholds of the plurality of thresholds.

19. The system according to claim 16, wherein the processor is further configured to:

select a buffer queue of the plurality of buffer queues from which to perform eviction based upon the evict policy, wherein the data is evicted from the selected buffer queue.