System having in-memory buffer service, temporary events file storage system and backup events file uploader service
First Claim
1. A system, comprising:
- a data store that is configured to store events; and
a temporary events file storage system (TEFSS);
a cluster of application servers each being configured to run a client application that generates events, wherein the cluster comprises;
a first application server comprising;
a capture service configured to;
receive events generated by the client application running on the first application server and to provide the events to an in-memory buffer service that is configured to temporarily hold the events in a bounded buffer that is implemented in physical memory at the first application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that are each configured to temporarily hold some of the events received from the capture service until a flush operation is performed, wherein each particular single-threaded buffer segment holds a plurality of events;
a consumer executor service configured to;
instantiate a plurality of consumer threads that each correspond to a particular single-threaded buffer segment, wherein each consumer thread comprises;
an instance of an events reader that is configured to read events flushed from a corresponding single-threaded buffer segment and to add an identifier to each the events; and
an instance of a direct events writer that is configured to attempt to directly write each event to the data store;
an indirect events writer that is invoked when events generated by the first application server are unable to be directly written to the data store, comprising;
a plurality of event capture threads each being configured to;
generate a particular events file that includes a plurality of events flushed from an in-memory buffer service, and write that particular events file to the TEFSS; and
a second application server comprising;
an events file uploader service that is configured to;
determine that the first application server is inactive;
read at least one events file from the TEFSS; and
write the events from each of the events files that was read to the data store.
1 Assignment
0 Petitions
Accused Products
Abstract
Computer-implemented methods and systems are provided. The system includes a data store that is configured to store events in an event table, a temporary events file storage system (TEFSS), and a cluster of application servers. The cluster includes a first application server that generates events, and a second application server that includes an events file uploader service. When the first application server is unable to directly write events to the data store, an indirect events writer generates events file(s), and writes the events file(s) to the TEFSS. Each events file includes a plurality of events flushed from an in-memory buffer service at the first application server. When the events file uploader service determines that the first application server is inactive, it reads the events file(s) from the TEFSS, and writes the events from each of the events files to the data store.
162 Citations
28 Claims
-
1. A system, comprising:
-
a data store that is configured to store events; and a temporary events file storage system (TEFSS); a cluster of application servers each being configured to run a client application that generates events, wherein the cluster comprises; a first application server comprising; a capture service configured to;
receive events generated by the client application running on the first application server and to provide the events to an in-memory buffer service that is configured to temporarily hold the events in a bounded buffer that is implemented in physical memory at the first application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that are each configured to temporarily hold some of the events received from the capture service until a flush operation is performed, wherein each particular single-threaded buffer segment holds a plurality of events;a consumer executor service configured to;
instantiate a plurality of consumer threads that each correspond to a particular single-threaded buffer segment, wherein each consumer thread comprises;
an instance of an events reader that is configured to read events flushed from a corresponding single-threaded buffer segment and to add an identifier to each the events; and
an instance of a direct events writer that is configured to attempt to directly write each event to the data store;an indirect events writer that is invoked when events generated by the first application server are unable to be directly written to the data store, comprising; a plurality of event capture threads each being configured to;
generate a particular events file that includes a plurality of events flushed from an in-memory buffer service, and write that particular events file to the TEFSS; anda second application server comprising; an events file uploader service that is configured to;
determine that the first application server is inactive;
read at least one events file from the TEFSS; and
write the events from each of the events files that was read to the data store. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
-
-
18. A non-transitory computer-readable medium including instructions, which when executed by a processing system at an application server having an in-memory buffer service, are configurable to cause the processing system to perform a method, comprising:
-
receiving, at a capture service at the application server, events from a client application running on the application server, wherein the events are to be stored in a data store; storing, via the capture service, the events in the in-memory buffer service comprising;
a bounded buffer that is implemented in physical memory at the application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that are each configured to temporarily store some of the events until a flush operation is performed;flushing the plurality of single-threaded buffer segments of the bounded buffer, wherein each particular single-threaded buffer segment includes the plurality of events; instantiating, via a consumer executor service at the application server, a plurality of consumer threads that each corresponds to a particular single-threaded buffer segment, wherein each consumer thread comprises;
an instance of an events reader and an instance of a direct events writer;reading, at the events reader of each consumer thread, the events flushed from the particular single-threaded buffer segment that corresponds to that consumer thread; adding, at each direct events writer, an identifier to each the events; at each direct events writer, attempting to directly write each event to the data store; instantiating, at an indirect events writer, a plurality of event capture threads each comprising an events file writer; generating, at each events file writer, a particular events file that comprises a plurality of events received from the in-memory buffer service when those events are unable to be directly written to the data store; and writing, via each events file writer, the particular events file to a temporary events file storage system (TEFSS) that is configured to temporarily store the events files for subsequent writing to the data store. - View Dependent Claims (19, 20)
-
-
21. A non-transitory computer-readable medium including instructions, which when executed by a processing system at a first application server, are configurable to cause the processing system to perform a method for storing events in a data store comprising an events uploader job detail table, wherein the first application server comprises:
- an events file uploader service that includes an events uploader manager, and the method comprising;
determining that a second application server is inactive, wherein determining that the second application server is inactive, comprises;
finding, via the events uploader manager, the second application server in the cluster that has been inactive for a certain duration;reading one or more events files from a temporary events file storage system (TEFSS), wherein each events file was written to the TEFSS by an indirect events writer of the second application server, and wherein each events file comprises a plurality of events, wherein the step of reading one or more events files from the TEFSS, comprises; reading, from an events uploader job detail table via the events uploader manager, a most recent uploader job record for the second application server; searching, via the events uploader manager, the most recent uploader job record for the second application server for a starting group identifier that uniquely identifies a group of events files generated by the second application server during a particular time window; and determining, via the events uploader manager, from the most recent uploader job record whether a status for the most recent uploader job record indicates running or completed; and writing each event from each of the events files that were read to the data store. - View Dependent Claims (22, 23, 24, 25, 26, 27)
- an events file uploader service that includes an events uploader manager, and the method comprising;
-
28. A computer-implemented method, comprising:
-
at a first application server; receiving, at a capture service, the events generated by the client application running on the first application server and providing the events to an in-memory buffer service, wherein the in-memory buffer service is configured to temporarily hold the events in a bounded buffer that is implemented in physical memory at the first application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that are each configured to temporarily hold some of the events received from the capture service until a flush operation is performed, wherein each particular single-threaded buffer segment holds a plurality of events; instantiating, via a consumer executor service, a plurality of consumer threads that each correspond to a particular single-threaded buffer segment, wherein each consumer thread comprises;
an instance of an events reader that is configured to read events flushed from a corresponding single-threaded buffer segment and to add an identifier to each the events; and
an instance of a direct events writer that is configured to attempt to directly write each event to a data store; andinvoking, an indirect events writer when events generated by the client application running on the first application server are unable to be directly written to a data store, wherein the indirect events writer comprises;
a plurality of event capture threads each comprising an instance of an events file writer;generating, at each events file writer of each event capture thread, a particular events file that comprises a plurality of events received from the in-memory buffer service when those events are unable to be directly written to the data store; writing, via each events file writer of each event capture thread, the particular events file generated by that event capture thread to a temporary events file storage system (TEFSS) that is configured to temporarily store one or more events files for subsequent writing to the data store; and at an events file uploader service of a second application server; determining that the first application server is inactive; reading at least one events file from the TEFSS that was written by the indirect events writer of the first application server; and writing the plurality of events from each events file that was read to the data store.
-
Specification