Techniques for more efficient generation of XML events from XML data sources

• US 8,543,898 B2
• Filed: 11/09/2007
• Issued: 09/24/2013
• Est. Priority Date: 11/09/2007
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Idicula, Sam, Zhang, Ning, Chandrasekar, Sivasankaran, Agarwal, Nipun
• Primary Examiner(s): STORK, KYLE R
• Application Number:
  US11/938,020
• Publication Number:
  US 20090125693A1
• Time in Patent Office: 2,146 Days
• Field of Search: 715/234, 715/243, 715/254, 715/255, 715/200
• US Class Current: 715/200
• CPC Class Codes: G06F 16/8373 Query execution
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/221 Parsing markup language streams streaming G06F40/149

Abstract

One may increase the efficiency of an XML event-generating process by reducing the number of requests to allocate or deallocate system memory. Such reduction may occur as the result of pre-allocating a memory chunk of sufficient size to contain all of the memory buffers required by a particular event-generating process. Instead of allocating new memory chunks for new memory buffers, an application may store any required buffers within the pre-allocated memory chunk. A sufficient memory size may be estimated by performing the event-generating process on a training set of XML documents. Also, an application may re-use buffers during the process or between different iterations of the process, thus avoiding the need to deallocate and reallocate memory that is essentially being used for the same purpose.

Claims

1. A method comprising:

based on one or more characteristics of an XML data source, determining a memory size for an XML event-generating process to process said XML data source;

prior to processing the XML data source using the XML event-generating process, requesting, from a memory manager, allocation of a memory chunk of the memory size for use with the XML event-generating process;

processing the XML data source using the XML event-generating process, thereby generating one or more XML events, wherein processing the XML data source comprises;

the XML event-generating process determining that processing the XML data source requires the use of a plurality of memory buffers; and

the XML event-generating process creating the plurality of memory buffers in the previously allocated memory chunk, without calling the memory manager; and

after processing the XML data source, requesting that the memory manager deallocate the memory chunk;

wherein the method is performed by one or more computing devices.

2. The method of claim 1,

wherein the XML data source is a binary-encoded XML document, wherein processing the XML data source further comprises decoding the binary-encoded XML document;

wherein determining to process the XML data source and processing the XML data source occur in response to a request from an application for textual form of the binary-encoded XML document.

3. The method of claim 1, wherein no other memory buffer is used during the processing of the XML data source.

4. The method of claim 1,

wherein the XML event-generating process is associated with a state machine representing an XPath expression, wherein the one or more XML events are used as input for the state machine;

wherein requesting the allocation of the memory chunk occurs during compilation of the state machine from the XPath expression.

5. The method of claim 1, further comprising:

prior to determining the memory size, executing the XML event-generating process in a plurality of iterations on a set of XML data sources;

while executing the XML event-generating process in the plurality of iterations, monitoring the XML event-generating process to determine, for each iteration of the plurality of iterations, a maximum total memory size of all memory buffers used during the iteration; and

wherein determining the memory size is based at least on the determined maximum total memory sizes.

6. The method of claim 5, wherein the memory size is an average of each of the determined maximum total memory sizes plus three times a standard deviation in the maximum total memory sizes over the plurality of iterations.

7. The method of claim 1, further comprising:

identifying a set of XML documents that are based on a particular XML schema;

determining the memory size based on executing the XML event-generating process in a plurality of iterations on a set of XML data sources, wherein the memory size is specific to the particular XML schema; and

determining that the memory chunk is to be of the memory size based on determining that the XML data source is also based on the particular XML schema.

8. The method of claim 1, wherein the memory manager is provided by one of:

a database system or an operating system.

9. The method of claim 1, wherein the memory size is specific to a particular XML schema, the method further comprising:

determining that the memory chunk is to be of the memory size based on determining that the XML data source is based on the particular XML schema.

10. The method of claim 1, further comprising, after processing the XML data source, adjusting the memory size for future data sources based upon memory usage while processing of the XML data source.

11. A method comprising:

based on one or more characteristics of a plurality of XML data sources, determining a memory size for an XML event-generating process to process said plurality of XML data sources;

prior to processing the plurality of XML data sources using the XML event-generating process, requesting, from a memory manager, allocation of a memory chunk of the memory size for use with the XML event-generating process;

processing the plurality of XML data sources using the XML event-generating process, thereby generating one or more XML events, wherein processing the plurality of XML data sources comprises;

wherein, for each particular XML data source of the plurality of XML data sources, processing the particular XML data source requires the use of at least one memory buffer; and

the XML event-generating process creating without calling the memory manager, each memory buffer required for processing the plurality of XML data sources in the same previously allocated memory chunk;

upon completion of processing the plurality of XML data sources, requesting that the memory manager deallocate the memory chunk;

wherein the method is performed by one or more computing devices.

12. The method of claim 11, wherein the memory size is specific to a particular XML schema, wherein each of the plurality of XML data sources pertains to the same particular XML schema.

13. The method of claim 11, wherein each of the plurality of XML data sources matches a particular query.

14. The method of claim 11, wherein processing the plurality of XML data sources further comprises:

processing a first XML data source, wherein processing the first XML data source comprises creating a first memory buffer stored in the memory chunk; and

processing a second XML data source, wherein processing the second XML data source comprises using the first memory buffer.

15. The method of claim 11, wherein processing the plurality of XML data sources further comprises processing a first XML data source, wherein processing the first XML data source comprises determining that the memory chunk has no room to store a particular memory buffer and, in response, requesting from the memory manager allocation of a temporary memory chunk to store the particular memory buffer.

16. The method of claim 11, wherein the memory size is a first size, the memory chunk is a first memory chunk, the plurality of XML data sources is a first plurality of XML documents, and the XML event-generating process is a first XML event-generating process;

wherein the first memory chunk is designated for use with documents that conform to a first XML schema, wherein the first plurality of XML documents conform to the first XML schema;

wherein the method further comprises, while maintaining the allocated first memory chunk, allocating a second memory chunk of a second size for use with a second XML event-generating process for processing a second plurality of XML documents that conform to a second XML schema, wherein the second XML schema is different than the first XML schema, wherein the second size is different than the first size.

17. The method of claim 11, further comprising determining the memory size based upon one or more of:

memory usage during execution of a training set of XML documents, an XML schema, an average tree depth, or a user-specified value.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause performance of steps comprising:

based on one or more characteristics of an XML data source, determining a memory size for an XML event-generating process to process said XML data source;

prior to processing the XML data source using the XML event-generating process, requesting, from a memory manager, allocation of a memory chunk of the memory size for use with the XML event-generating process;

processing the XML data source using the XML event-generating process, thereby generating one or more XML events, wherein processing the XML data source comprises;

the XML event-generating process determining that processing the XML data source requires the use of a plurality of memory buffers; and

the XML event-generating process creating the plurality of memory buffers in the previously allocated memory chunk, without calling the memory manager; and

after processing the XML data source, requesting that the memory manager deallocate the memory chunk.

19. The one or more non-transitory computer-readable media of claim 18,

wherein the XML data source is a binary-encoded XML document, wherein processing the XML data source further comprises decoding the binary-encoded XML document;

wherein determining to process the XML data source and processing the XML data source occur in response to a request from an application for textual form of the binary-encoded XML document.

20. The one or more non-transitory computer-readable media of claim 18, wherein no other memory buffer is used during the processing of the XML data source.

21. The one or more non-transitory computer-readable media of claim 18,

wherein the XML event-generating process is associated with a state machine representing an XPath expression, wherein the one or more XML events are used as input for the state machine;

wherein requesting the allocation of the memory chunk occurs during compilation of the state machine from the XPath expression.

22. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising:

prior to determining the memory size, executing the XML event-generating process in a plurality of iterations on a set of XML data sources;

while executing the XML event-generating process in the plurality of iterations, monitoring the XML event-generating process to determine, for each iteration of the plurality of iterations, a maximum total memory size of all memory buffers used during the iteration; and

wherein determining the memory size is based at least on the determined maximum total memory sizes.

23. The one or more non-transitory computer-readable media of claim 22, wherein the memory size is an average of each of the determined maximum total memory sizes plus three times a standard deviation in the maximum total memory sizes over the plurality of iterations.

24. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising:

identifying a set of XML documents that are based on a particular XML schema;

determining the memory size based on executing the XML event-generating process in a plurality of iterations on a set of XML data sources, wherein the memory size is specific to the particular XML schema; and

determining that the memory chunk is to be of the memory size based on determining that the XML data source is also based on the particular XML schema.

25. The one or more non-transitory computer-readable media of claim 18, wherein the memory manager is provided by one of:

a database system or an operating system.

26. The one or more non-transitory computer-readable media of claim 24, wherein the memory size is specific to the particular XML schema, wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising:

determining that the memory chunk is to be of the memory size based on determining that the XML data source is based on the particular XML schema.

27. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising:

after processing the XML data source, adjusting the memory size for future data sources based upon memory usage while processing of the XML data source.

28. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause performance of steps comprising:

based on one or more characteristics of a plurality of XML data sources, determining a memory size for an XML event-generating process to process said plurality of XML data sources;

prior to processing the plurality of XML data sources using the XML event-generating process, requesting, from a memory manager, allocation of a memory chunk of the memory size for use with the XML event-generating process;

processing the plurality of XML data sources using the XML event-generating process, thereby generating one or more XML events, wherein processing the plurality of XML data sources comprises;

wherein, for each particular XML data source of the plurality of XML data sources, processing the particular XML data source requires the use of at least one memory buffer; and

the XML event-generating process storing creating each memory buffer required for processing the plurality of XML data sources in the same previously allocated memory chunk;

upon completion of processing the plurality of XML data sources, requesting that the memory manager deallocate the memory chunk.

29. The one or more non-transitory computer-readable media of claim 28, wherein the memory size is specific to a particular XML schema, wherein each of the plurality of XML data sources pertains to the same particular XML schema.

30. The one or more non-transitory computer-readable media of claim 28, wherein each of the plurality of XML data sources matches a particular query.

31. The one or more non-transitory computer-readable media of claim 28, wherein processing the plurality of XML data sources further comprises:

processing a first XML data source, wherein processing the first XML data source comprises creating a first memory buffer stored in the memory chunk; and

processing a second XML data source, wherein processing the second XML data source comprises using the first memory buffer.

32. The one or more non-transitory computer-readable media of claim 28, wherein processing the plurality of XML data sources further comprises processing a first XML data source, wherein processing the first XML data source comprises determining that the memory chunk has no room to store a particular memory buffer and, in response, requesting from the memory manager allocation of a temporary memory chunk to store the particular memory buffer.

33. The one or more non-transitory computer-readable media of claim 28, wherein the memory size is a first size, the memory chunk is a first memory chunk, a plurality of documents is a first plurality of XML documents, and the XML event-generating process is a first XML event-generating process;

wherein the first memory chunk is designated for use with documents that conform to a first XML schema, wherein the first plurality of XML documents conform to the first XML schema;

wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising, while maintaining the allocated first memory chunk, allocating a second memory chunk of a second size for use with a second XML event-generating process for processing a second plurality of XML documents that conform to a second XML schema, wherein the second XML schema is different than the first XML schema, wherein the second size is different than the first size.

34. The one or more non-transitory computer-readable media of claim 28, wherein the instructions, when executed by the one or more computing devices, further cause performance of steps comprising:

determining the memory size based upon one or more of;

memory usage during execution of a training set of XML documents, an XML schema, an average tree depth, or a user-specified value.