Processing of data stream collection record sequence

• US 9,766,929 B2
• Filed: 04/14/2015
• Issued: 09/19/2017
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Diaconu, Cristian, Vasquez Lopez, Daniel, Kodandaramaih, Raghavendra Thallam, Brjazovski, Arkadi, Ramos, Rogerio
• Primary Examiner(s): Kim, Sisley
• Application Number:
  US14/686,525
• Publication Number:
  US 20160306659A1
• Time in Patent Office: 889 Days
• Field of Search: None
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/54 Interprogram communication

Abstract

The use of a data stream that has therein data items and a sequence of collection records. each comprising a collection definition that is not overlapping with the collection definition in any of the sequence of collection records. The collection definition defines which data items of the data stream are included within the collection. Each collection record also includes a data stream address range at least extending from the data stream address of the first data item of the collection to the data stream address of the last data item in the collection. In this context, the data stream may be efficiently processed by quickly reviewing the data stream to find each collection record. Once a collection record is found, the collection record is dispatched for processing to a worker thread for processing of the corresponding collection.

Claims

1. A computing system, comprising:

one or more processors;

system memory; and

a data stream review component that is configured to;

identify a data stream that includes a plurality of data items;

identify a first collection within the data stream, wherein the first collection includes a first set of data items that each share one or more common properties, the one or more common properties being defined by a collection definition that is associated with the first collection;

identify a second set of data items that are of a first type but that do not share the one or more common properties;

modify the collection definition to include the first type, whereby, after the collection definition is modified, the second set of data items is included within the first collection;

identify a collection record of the first collection, wherein the collection record was created after the collection definition was modified, and wherein the collection record was inserted into the data stream following a last data item included within the first collection, the collection record being inserted into the data stream at least partially in response to

     1) determining that one or more thresholds of the first collection have been reached and

     2) detecting the last data item of the first collection; and

dispatch the first collection record to a worker thread for processing.

2. The computing system in accordance with claim 1, the data stream being a log.

3. The computing system in accordance with claim 2, the data items of the data stream including task entries.

4. The computing system in accordance with claim 3, the log being a transactional log such that the task entries may belong to a transaction.

5. The computing system in accordance with claim 3, wherein the collection definition comprises a transaction identifier.

6. The computing system in accordance with claim 3, wherein the collection definition comprises a transaction identifier set.

7. The computing system in accordance with claim 3, wherein the collection definition comprises a transaction identifier range.

8. The computing system in accordance with claim 1, wherein the collection record includes the collection definition, wherein a second collection record corresponding to a second collection includes a second collection definition, and wherein the first collection definition does not overlap the second collection definition.

9. The computing system in accordance with claim 1, wherein the collection record further includes a data stream address range.

10. The computing system in accordance with claim 1, wherein after receiving the collection record, a first worker thread processes the first collection by serializing a portion of the data stream represented by a data stream address range that is included within the collection record.

11. The computing system in accordance with claim 10, wherein the first worker thread performs a serialization in preparation for persisting the portion of the data stream.

12. The computing system in accordance with claim 11, wherein the first worker thread also determines a target file to persist the portion of the data stream in.

13. The computing system in accordance with claim 12, wherein the target file depends on the data stream address range included in the collection record.

14. The computing system in accordance with claim 10, wherein the first worker thread serializes in preparation for transmission of the portion of the data stream.

15. A method, implemented at a computer system that includes one or more processors, for processing information included within a data stream, the method comprising:

identifying a data stream that includes a plurality of data items;

identifying a first collection within the data stream, wherein the first collection includes a first set of data items that each share one or more common properties, the one or more common properties being defined by a collection definition that is associated with the first collection;

identifying a second set of data items that are of a first type but that do not share the one or more common properties;

modifying the collection definition to include the first type, whereby, after the collection definition is modified, the second set of data items is included within the first collection;

identifying a collection record of the first collection, wherein the collection record was created after the collection definition was modified, and wherein the collection record was inserted into the data stream following a last data item included within the first collection, the collection record being inserted into the data stream at least partially in response to

     1) determining that one or more thresholds of the first collection have been reached and

     2) detecting the last data item of the first collection; and

dispatching the first collection record to a worker thread for processing.

16. The method in accordance with claim 15, the data stream being a log, the data items of the data stream including task entries.

17. The method in accordance with claim 16, the log being a transactional log such that the task entries may belong to a transaction.

18. The method in accordance with claim 16, wherein the collection definition comprises a transaction identifier.

19. The method in accordance with claim 16, wherein the collection definition comprises a transaction identifier set.

20. The method in accordance with claim 16, wherein the collection definition comprises a transaction identifier range.

21. The method in accordance with claim 20, wherein, after receiving the collection record, a first worker thread processes the first collection by redoing a portion of the log within a corresponding data stream address range that is included within the collection record.

22. The method in accordance with claim 20, wherein a first worker thread performs redo of tasks within a data stream address range included within the t collection record, the data stream address range corresponding to transactions within the transaction identifier range of the collection record.

23. The method in accordance with claim 15, wherein a first worker thread processes the first collection by serializing a portion of the data stream represented in a data stream address range that is included within the collection record.

24. A method in accordance with claim 17, wherein a reference count is maintained, the reference count initially detailing how many transactions have not yet been completed, and wherein the method further comprises:

decrementing the reference count each time one of the transactions is completed.

25. A computer program product comprising one or more computer-readable hardware storage devices having stored thereon computer-executable instructions that are executable by one or more processors of the computing system to cause the computing system to process information included within a data stream by at least causing the computing system to:

identify a data stream that includes a plurality of data items;

identify a first collection within the data stream, wherein the first collection includes a first set of data items that each share one or more common properties, the one or more common properties being defined by a collection definition that is associated with the first collection;

identify a second set of data items that are of a first type but that do not share the one or more common properties;

modify the collection definition to include the first type, whereby, after the collection definition is modified, the second set of data items is included within the first collection;

identify a collection record of the first collection, wherein the collection record was created after the collection definition was modified, and wherein the collection record was inserted into the data stream following a last data item included within the first collection, the collection record being inserted into the data stream at least partially in response to

     1) determining that one or more thresholds of the first collection have been reached and

     2) detecting the last data item of the first collection; and

dispatch the first collection record to a worker thread for processing.

26. The computer program product in accordance with claim 25, wherein a first worker thread processes the first collection by serializing a portion of the data stream represented in a data stream address range that is included within the collection record.

27. The computer program product in accordance with claim 26, wherein the data stream is a log, and wherein the first worker thread serializes in preparation for persisting a portion of the log.

28. The computer program product in accordance with claim 27, wherein the first worker thread also determines a target file to persist the portion of the log in.

29. The computer program product in accordance with claim 28, wherein the target file depends on the data stream address range represented by the collection record dispatched to the worker thread.

30. The computer program product in accordance with claim 25, wherein the data stream is a log, and wherein a first worker thread serializes in preparation for transmission of a portion of the log.