Method and apparatus of streaming data transformation using code generator and translator

• US 7,590,644 B2
• Filed: 02/28/2005
• Issued: 09/15/2009
• Est. Priority Date: 12/21/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Morgan, Charles Robert, Kenton, Jeffrey, Matsakis, Nicholas, Nelson, Jan-Christian, Kuznetsov, Eugene, Dolph, Cyrus Abda V
• Primary Examiner(s): Mofiz; Apu M
• Assistant Examiner(s): STACE, BRENT S
• Application Number:
  US11/068,369
• Publication Number:
  US 20050273772A1
• Time in Patent Office: 1,660 Days
• Field of Search: 707/3, 707/10, 707100-102, 715235-239
• US Class Current: 1/1
• CPC Class Codes: G06F 8/41 Compilation

Abstract

A high level transformation method and apparatus for converting data formats in the context of network applications, among other places. A flexible transformation mechanism is provided that facilitates generation of translation machine code. A translator is dynamically generated by a translator compiler engine. When fed an input stream, the translator generates an output stream by executing the native object code generated on the fly by the translator compiler engine. In addition, the translator may be configured to perform a bi-directional translation between the two streams as well as translation between two distinct protocol sequences. Further a translator may working in streaming mode, to facilitate streaming processing of documents.

Claims

1. A method of processing XML data, said method comprising:

consuming a portion of an XML-encoded input data stream;

processing said consumed portion of an XML-encoded data stream based upon a transformation definition;

compiling at least one XSLT stylesheet, including performing an analysis of said at least one XSLT stylesheet, to determine whether at least some XML data to be processed in accordance with said at least one XSLT stylesheet can be processed in a streaming fashion;

outputting result data produced as a result of processing said consumed portion of an XML-encoded data stream on a processor according to said transformation definition before all of said XML-encoded input data stream has been consumed upon determining that the consumed portion of the XML-encoded data stream can be processed in a streaming fashion;

consuming the entire XML-encoded data stream and processing the consumed XML-encoded data stream on a processor using a non-streaming model upon determining that the consumed portion of the XML-encoded data stream cannot be processed in a streaming fashion.

2. The method of claim 1 wherein said step of processing said consumed portion of an XML-encoded data stream based upon a transformation definition includes producing side effects.

3. The method of claim 2 wherein said side effects include one of routing data, effecting a direction of flow of messages, or selection of subsequent processing steps.

4. The method of claim 1 wherein said transformation definition includes XML transforms, XSLT stylesheets, XPath expressions, XQuery, XACML, XML encryption, XML digital signature, web services security processing and XML Schemas.

5. The method of claim 1 wherein said step of performing analysis of said at least one XSLT stylesheet, to determine whether at least some XML data to be processed in accordance with said at least one XSLT stylesheet can be processed in a streaming fashion includes determining whether said at least one XSLT stylesheet specifies access to data that is not readily available at a time of such access;

and

wherein a streaming fashion comprises outputting result data before the XML-encoded input data stream is consumed.

6. The method of claim 1 wherein said step of outputting result data produced as a result of processing said consumed portion of an XML-encoded data stream according to said stylesheet, includes buffering at least some result data in an out of order buffer to maintain a proper order of outputting result data.

22. The method as in claim 1, further comprising:

maintaining a cache for storing at least one XML document required by the XSLT stylesheet to perform said transformation definition, the cache comprising a fixed size table indexed according to a uniform resource locator associated with the at least one XML document.

23. The method as in claim 22, wherein storing the at least one XML document includes:

parsing the at least one XML document into a tree representing at least a portion of the at least on XML document.

24. The method as in claim 23, wherein storing the at least one XML document includes:

when the fixed size table of the cache is full upon attempting to the store the at least one XML document in the cache;

discarding a least recently used entry in the tree;

associating a newly added entry in the tree with at least a portion of XML data from the at least one XML document;

creating a first reference to be associated with the newly-added entry;

creating a second reference to be associated with at least one transformation activity that is currently using the at least one XML document.

25. The method as in claim 24, wherein discarding a least recently used entry in the tree includes:

determining a reference count associated with the discarding a least recently used entry in the tree is equal to zero.

26. The method of claim 1 wherein the non-streaming model comprises a document object model.

27. The method of claim 1 wherein the determination as to whether at least some XML data to be processed in accordance with said at least one XSLT stylesheet can be processed in a streaming fashion is made in conjunction with compiling the at least one XSLT stylesheet.

7. A method for transforming XML documents in streaming mode, said method comprising:

consuming a portion of an XML-encoded input data stream;

processing said consumed portion of an XML-encoded data stream based upon a transformation definition;

compiling at least one XSLT stylesheet, including performing an analysis of said at least one XSLT stylesheet related to at least one transformation definition, to determine whether at least some XML data to be processed in accordance with said at least one XSLT stylesheet can be processed in a streaming fashion;

generating executable machine instructions based on said step of compiling at least one XSLT stylesheet upon determining that the portions of the XML documents are streamable, said executable machine instructions to transform XML documents;

executing said executable machine instructions with aid of a streaming runtime process; and

processing the XML documents on a processor using a document object model upon determining that the portions of the XML documents are not streamable.

8. The method of claim 7 wherein said step of analyzing said at least one transformation definition to determine if portions of it are streamable includes determining whether said at least one transformation definition specifies access to data that is not readily available at a time of such access.

9. The method of claim 7 wherein said step of compiling at least one XSLT stylesheet includes applying at least one optimization to generate optimized executable machine instructions.

10. The method of claim 7 where said XSLT stylesheet includes web services security processing.

11. The method of claim 7 where said compilation includes breaking up said XSLT stylesheet into multiple strands.

12. The method of claim 11 where said executable machine instructions making up the strands are executed on different processors.

13. The method of claim 9 wherein said at least one optimization includes common subexpression elimination, constant propagation, code motion, loop unrolling, strength reduction, peephole optimization, dead code elimination, LRE Folding, type resolution and tightening, adaptive profile-guided optimization, template or method specialization, and inlining.

14. The method of claim 7 wherein said transformation definition includes XML transforms, XSLT stylesheets, XPath expressions, XQuery definitions, XACML, XML encryption, XML digital signatures, and XML Schemas.

15. The method of claim 7 wherein transforming XML documents in streaming mode includes buffering at least some result data to maintain a proper order of outputting result data.

16. A network appliance device for performing streaming XML processing, comprising:

an input port to consume a portion of XML-encoded input data file;

executable machine instructions generated by compiling a XSLT stylesheet, wherein said compilation includes analyzing a transformation definition to determine whether XML-encoded data processed by a processor in accordance with said transformation definition can be processed in streaming mode, said executable machine instructions coupled to said input port, to process said portion of an XML-encoded input data file;

wherein upon determining that the portion XML-encoded input data file can be processed in streaming mode, said executable machine instructions processes said portion of an XML-encoded data file outputs result data before all of said XML-encoded input data file has been consumed by said input port.

17. The network appliance device of claim 16 further including at least one output buffer, said output receiving result data from said executable machine instructions, and wherein said output buffer holds result data until after other result data has been outputted by said executable machine instructions.

18. The network appliance device of claim 16 wherein said transformation definition includes XML transforms, XSLT stylesheets, XPath expressions, XQuery, XACML, XML Digital Signatures, XML Encryption, web services security processing and XML Schemas.

19. The network appliance device of claim 16 which serves primarily as a web services security gateway (also known in the industry as XML firewall or XML security gateway).

20. An apparatus for streaming mode XML processing, said apparatus comprising:

a compiler configured to compile at least one XSLT stylesheet and to generate executable machine instructions from XML processing definition files, said executable machine instructions to process input XML data in streaming mode;

an analyzer, operating in conjunction with the compiler, configured to perform an analysis of at least one XSLT stylesheet related to at least one transformation definition to determine if the input XML data can be processed in streaming mode;

a push-mode XML parser configured to produce parse trees or parse events based on said input XML data; and

a parse event distributor, to receive data from said push-mode XML parser and an out-of-order output buffering system to buffer output from processing said XML input data; and

a processor to output result data produced as a result of processing said XML-encoded data stream according to said XSLT stylesheet before all of said XML-encoded input data stream has been processed upon determining that the processed portion of the XML-encoded data stream can be processed in a streaming fashion.

21. A computer implemented method for processing of XML data, comprising:

receiving a portion of an encoded XML-encoded input data stream;

applying a transformation definition to the portion of the encoded XML-encoded input data stream, which includes executing a streaming compiling sequence, the streaming compiling sequence performing an analysis of at least one XSLT stylesheet related to the transformation definition in order to validate that the data from the portion of the encoded XML-encoded input data stream can be processed with respect to the XSLT stylesheet and can be stream processed on the fly;

outputting an output data stream subsequent to the streaming compiling sequence upon validating that the data from the portion of the encoded XML-encoded input data stream can be stream processed on the fly, the output data stream representing the portion of the encoded XML-encoded input data stream transformed according to the transformation definition, the output data stream created before the encoded XML-encoded input data stream is received in its entirety; and

processing the consumed XML-encoded data stream on a processor using a document object model upon validating that the data from the portion of the encoded XML-encoded input data stream cannot be stream processed on the fly.