System having in-memory buffer service, temporary events file storage system and backup events file uploader service

• US 10,185,603 B2
• Filed: 04/05/2018
• Issued: 01/22/2019
• Est. Priority Date: 09/15/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Pradeep, Aakash, Torman, Adam, Warshavsky, Alex, Jain, Samarpan
• Primary Examiner(s): Dorais, Craig C
• Application Number:
  US15/946,596
• Publication Number:
  US 20180225159A1
• Time in Patent Office: 292 Days
• Field of Search: None
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1004 Server selection for load balancing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Computer-implemented methods and systems are provided. The system includes a data store that is configured to store events in an event table, a temporary events file storage system (TEFSS), and a cluster of application servers. The cluster includes a first application server that generates events, and a second application server that includes an events file uploader service. When the first application server is unable to directly write events to the data store, an indirect events writer generates events file(s), and writes the events file(s) to the TEFSS. Each events file includes a plurality of events flushed from an in-memory buffer service at the first application server. When the events file uploader service determines that the first application server is inactive, it reads the events file(s) from the TEFSS, and writes the events from each of the events files to the data store.

Claims

1. A system, comprising:

a data store comprising memory that is configured to store events; and

a temporary events file storage system (TEFSS);

a cluster of application servers each comprising one or more hardware-based processors, wherein each of the application servers are configured to run a client application that generates events, wherein the cluster comprises;

a first application server comprising one or more hardware-based processors configured to execute;

an indirect events writer that is invoked when events generated by the first application server are unable to be directly written to the data store, comprising;

a plurality of event capture threads each being configured to;

generate at least one events file and write the at least one events file to the TEFSS, wherein each events file includes a plurality of events flushed from an in-memory buffer service; and

a second application server comprising one or more hardware-based processors configured to execute;

an events file uploader service that is configured to;

determine that the first application server is inactive;

read at least one events file from the TEFSS; and

write the events from each of the events files that was read to the data store.

2. The system according to claim 1, wherein the one or more hardware-based processors of the first application server are further configured to execute:

a capture service configured to receive the events generated by the client application running on the the first application server and to provide the events to the in-memory buffer service,

wherein the in-memory buffer service temporarily holds the events in a bounded buffer that is implemented in physical memory at the first application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that each temporarily hold some of the events received from the capture service until a flush operation is performed, wherein each particular single-threaded buffer segment holds a plurality of events;

a consumer executor service configured to;

instantiate a plurality of consumer threads that each correspond to a particular single-threaded buffer segment, wherein each consumer thread comprises;

an instance of an events reader that is configured to read events flushed from the corresponding single-threaded buffer segment and to add an identifier to each the events; and

an instance of a direct events writer that is configured to attempt to directly write each event to the data store.

3. The system according to claim 2, wherein each consumer thread provides the events to a corresponding one of the event capture threads, and wherein each of the event capture threads comprises:

an instance of an events file writer; and

an instance of an invoker that is configured to;

determine whether a corresponding instance of the direct events writer was able to successfully directly write each event, that was read from the corresponding single-threaded buffer segment, to an event table at the data store; and

invoke the instance of the events file writer at that event capture thread, whenever the invoker determines that the corresponding instance of the direct events writer was unable to successfully write each event directly to the data store, to serialize the plurality of events from the particular single-threaded buffer segment to generate an events file and to write the events file to the TEFSS.

4. The system according to claim 1, wherein each events file comprises:

a file name;

a group identifier that uniquely identifies a group of events files that were generated during a particular time window; and

file information comprising a file type, and a plurality of events that are serialized.

5. The system according to claim 4, wherein the file name comprises:

an application server name of the application server that generated the events in that events file;

a creation date for each of the events in that events file; and

a thread identifier that specifies a name of an event capture thread writing that events file.

6. The system according to claim 1, wherein the data store further comprises:

an events uploader job detail table that comprises a plurality of rows, and

wherein the events file uploader service comprises;

an events uploader manager that is configured to;

create an uploader job record for each events file stored at the TEFSS, wherein each uploader job record includes job detail information that points to that particular events file, and

write each uploader job record to one row of the events uploader job detail table maintained at the data store.

7. The system according to claim 6, wherein each uploader job record comprises:

a cluster identifier that uniquely identifies a collection of instances that are grouped as the cluster;

an instance identifier that uniquely identifies a collection of application servers grouped as an instance;

an application server name of a particular application server;

a group identifier that uniquely identifies a group of events files that were generated during a particular time window;

a job started date that specifies when that particular events file was created; and

a job type that indicates either regular or backup.

8. The system according to claim 7, wherein each uploader job record further comprises:

a file name of that particular events file, and

wherein each row of the events uploader job detail table includes a row key that is used by the events uploader manager to look up job detail information for that particular event file, wherein the row key is defined by the combination of the instance identifier, the application server name, the group identifier, the file name, and the job started date.

9. The system according to claim 7, wherein the cluster identifier, the instance identifier, the application server name, and the job type are used to identify a particular instance of the events file uploader service that runs on a particular application server of the cluster.

10. The system according to claim 7, wherein each uploader job record further comprises:

a number of events that specifies a number of events for that particular events file;

a last updated time that indicates when that particular events file was last updated;

a number of retry attempts that specifies how many times the events file uploader service has attempted to process that particular events file; and

a status that indicates either running or completed.

11. The system according to claim 1, wherein the events file uploader service further comprises:

an events uploader manager that is configured to;

maintain each of the uploader job records that are stored in an events uploader job detail table; and

retrieve job detail information from particular uploader job records stored at the events uploader job detail table of the data store.

12. The system according to claim 11, wherein the events uploader manager is further configured to:

find an application server in the cluster that has been inactive for a certain duration, wherein the events uploader manager determines that the first application server has been inactive for the certain duration;

read, from the events uploader job detail table, a most recent uploader job record for the first application server;

search the most recent uploader job record for a starting group identifier that uniquely identifies a group of events files generated by the first application server during a particular time window; and

determine from the most recent uploader job record whether a status for the most recent uploader job record indicates running or completed.

13. The system according to claim 12, wherein the events uploader manager is further configured to:

when the status of the most recent uploader job record indicates completed;

search the TFESS for an events file that includes the starting group identifier; and

determine whether the events file that is associated with the starting group identifier was able to be read from the TEFSS.

14. The system according to claim 13, when the events file that is associated with the starting group identifier was not able to be read from the TEFSS, wherein the events uploader manager is further configured to:

determine a next possible group identifier that uniquely identifies another group of events files generated by the first application server during another particular time window;

determine whether the next possible group identifier is for a current time window;

search, when the next possible group identifier is not for the current time window, the TFESS for an events file that includes the next possible group identifier, or

terminate the events file uploader service when the next possible group identifier is for the current time window.

15. The system according to claim 13, when the events file that includes the starting group identifier was able to be read from the TEFSS, wherein the events uploader manager is further configured to:

instantiate and assign a worker thread to each particular events file that was read, wherein each worker thread comprises;

an events file reader and an event loader; and

upsert the most recent uploader job record for the first application server with updated job detail information at the events uploader job detail table of the data store;

wherein each events file reader is configured to;

read an events file from the TEFSS based on the job detail information retrieved from the events uploader job detail table, serialize the events from that events file, and send the events to the event loader.

16. The system according to claim 15, wherein the event loader is configured to:

load and save events from each particular events file to a row of an event table maintained at the data store;

wherein the events uploader manager is further configured to;

update, in the events uploader job detail table, the status of each uploader job record to indicate completed upon loading and saving;

delete each of the particular events files from the TEFSS; and

search the TFESS for another events file that includes the starting group identifier.

17. The system according to claim 12, when it is determined from the most recent uploader job record that status for the most recent uploader job record indicates running, wherein the events uploader manager is further configured to:

assign a worker thread for each particular events file that was read, wherein each worker thread comprises;

an events file reader and an event loader; and

upsert the most recent uploader job record for the first application server with updated job detail information at the events uploader job detail table of the data store;

wherein the events file reader is configured to;

read the events file, serialize the events from the events file and send the events to the event loader.

18. The system according to claim 17, wherein the event loader is configured to:

load and save events from each particular events file to a row of an event table maintained at the data store;

wherein the events uploader manager is further configured to;

update, in the events uploader job detail table, the status of each uploader job record to indicate completed upon loading and saving;

delete each of the particular events files from the TEFSS; and

search the TFESS for another events file that includes the starting group identifier.

19. A non-transitory computer-readable medium including instructions, which when executed by a processing system at an application server having an in-memory buffer service, are configurable to cause the processing system to perform a method, comprising:

instantiating, at an indirect events writer, a plurality of event capture threads each comprising an events file writer;

generating, at each events file writer, a particular events file that comprises a plurality of events received from the in-memory buffer service when those events are unable to be directly written to a data store; and

writing, via each events file writer, the particular events file to a temporary events file storage system (TEFSS), wherein the TEFSS comprises memory that is configured to temporarily store each events file from each events file writer for subsequent writing to the data store,

wherein each events file comprises;

a file name comprising;

an application server name of the application server that generated the events in the events file;

a creation date for each of the events in the events file; and

a thread identifier that specifies a name of an event capture thread writing the events file;

a group identifier that uniquely identifies a group of events files that were generated during a particular time window; and

a plurality of events that are serialized.

20. The non-transitory computer-readable medium of claim 19, further comprising:

receiving, at a capture service at the application server, events from a client application running on the application server, wherein the events are to be stored in the data store;

storing, via the capture service, the events in the in-memory buffer service comprising;

a bounded buffer that is implemented in physical memory at the application server, wherein the bounded buffer comprises a plurality of single-threaded buffer segments that each temporarily store some of the events until a flush operation is performed;

flushing the plurality of single-threaded buffer segments of the bounded buffer, wherein each particular single-threaded buffer segment includes the plurality of events;

instantiating, via a consumer executor service at the application server, a plurality of consumer threads that each corresponds to a particular single-threaded buffer segment, wherein each consumer thread comprises;

an instance of an events reader and an instance of a direct events writer;

reading, at the events reader of each consumer thread, the events flushed from the particular single-threaded buffer segment that corresponds to that consumer thread;

adding, at each direct events writer, an identifier to each the events; and

at each direct events writer, attempting to directly write each event to the data store.

21. The non-transitory computer-readable medium of claim 20, wherein each of the event capture threads further comprises:

an invoker, and wherein the method further comprises;

determining, at each invoker, whether a corresponding instance of the direct events writer was able to successfully write the events, that were read from the particular single-threaded buffer segment, directly to an event table at the data store;

whenever an instance of an invoker at one of the event capture thread determines that a corresponding instance of the direct events writer was unable to successfully write the events directly to the data store, invoking an instance of the events file writer at that event capture thread, and receiving the plurality of events from that corresponding instance of the direct events writer for that events file writer; and

serializing, at each instance of the events file writer, the plurality of events from the particular single-threaded buffer segment into an events file.

22. A non-transitory computer-readable medium including instructions, which when executed by a hardware-based processing system at a first application server, are configurable to cause the hardware-based processing system to perform a method for storing events in a data store that comprises an events uploader job detail table that comprises a plurality of rows each being configured to store an uploader job record, the method comprising:

determining that a second application server is inactive, wherein the first application server and second application server are part of a cluster of application servers;

executing, at the first application server, an events file uploader service that includes an events uploader manager that maintains each of the uploader job records, wherein each uploader job record includes job detail information that points to a particular events file stored in the TEFSS, wherein each uploader job record comprises;

a cluster identifier that uniquely identifies a collection of instances that a grouped as a cluster;

an instance identifier that uniquely identifies a collection of application servers grouped as an instance;

an application server name of a particular application server; and

a group identifier that uniquely identifies a group of events files that were generated during a particular time window;

reading one or more events files from memory of a temporary events file storage system (TEFSS), wherein each events file was written to the TEFSS by an indirect events writer of the second application server, and wherein each events file comprises a plurality of events; and

writing, via the events file uploader service, each event from each of the events files that were read to the data store.

23. The non-transitory computer-readable medium of claim 22,

wherein determining that the second application server is inactive, comprises:

finding, via the events uploader manager of the first application server, an application server in a cluster that has been inactive for a certain duration, wherein the events uploader manager determines that the second application server has been inactive for the certain duration;

wherein the step of reading one or more events files from the TEFSS, comprises;

reading, from an events uploader job detail table via the events uploader manager of the first application server, a most recent uploader job record for the second application server;

searching, via the events uploader manager of the first application server, the most recent uploader job record for the second application server for a starting group identifier that uniquely identifies a group of events files generated by the second application server during a particular time window; and

determining, via the events uploader manager of the first application server, from the most recent uploader job record whether a status for the most recent uploader job record indicates running or completed.

24. The non-transitory computer-readable medium of claim 23, when the status of the most recent uploader job record indicates completed:

searching, via the events uploader manager of the first application server, the TFESS for an events file that includes the starting group identifier; and

determining, via the events uploader manager of the first application server, whether an events file that is associated with the starting group identifier was able to be read from the TEFSS.

25. The non-transitory computer-readable medium of claim 24, when the events file that is associated with the starting group identifier was not able to be read from the TEFSS:

determining, via the events uploader manager of the first application server, a next possible group identifier that uniquely identifies another group of events files generated by the second application server during another particular time window;

determining, via the events uploader manager of the first application server, whether the next possible group identifier is for a current time window;

when the next possible group identifier is for the current time window, terminating the events file uploader service at the first application server; and

when the next possible group identifier is not for the current time window, searching, via the events uploader manager of the first application server, the TFESS for an events file that includes the next possible group identifier.

26. The non-transitory computer-readable medium of claim 24, when the events file that includes the starting group identifier was able to be read from the TEFSS:

assigning, via the events uploader manager of the first application server, a worker thread for the events file that was read, wherein the worker thread comprises;

an events file reader and an event loader; and

upserting, via the events uploader manager of the first application server, the most recent uploader job record for the second application server at the events uploader job detail table of the data store with updated job detail information; and

at the events file reader;

reading at least one events file for the application server from the TEFSS based on the job detail information retrieved, serializing the events from the events file, and sending the events to the event loader;

wherein the step of writing each event from each of the events files that were read to the data store, comprises;

loading and saving, via the event loader, each of the events from the events file to a row of an event table maintained at the data store; and

further comprising;

updating in the events uploader job detail table, via the events uploader manager, the status of the uploader job record that corresponds to the events file to indicate completed;

deleting, via the events uploader manager, the events file from the TEFSS; and

searching, via the events uploader manager, the TFESS for another events file that includes the starting group identifier.

27. The non-transitory computer-readable medium of claim 23, when it is determined from the most recent uploader job record that status for the most recent uploader job record indicates running, further comprising:

assigning, via the events uploader manager, a worker thread for the events file that was read, wherein the worker thread comprises;

an events file reader and an event loader; and

upserting, at the events uploader job detail table of the data store via the events uploader manager, the most recent uploader job record for the second application server with updated job detail information; and

at the events file reader;

reading the events file that includes the starting group identifier, serializing the events from the events file and sending the events to the event loader;

wherein the step of writing the plurality of events from each events file that was read to an event table maintained at the data store, comprises;

loading and saving, via the event loader, each of the events from the events file to a row of the event table maintained at the data store; and

further comprising;

updating, in the events uploader job detail table via the events uploader manager, the status of the uploader job record that corresponds to the events file to indicate completed;

deleting, via the events uploader manager, the events file from the TEFSS; and

searching, via the events uploader manager, the TFESS for another events file that includes the starting group identifier.

28. A computer-implemented method, comprising:

at a first application server comprising one or more hardware-based processors configured to execute;

an in-memory buffer service, and an indirect events writer that is configured to instantiate a plurality of event capture threads each comprising an instance of an events file writer;

generating, at each events file writer of each event capture thread, a particular events file that comprises a plurality of events received from the in-memory buffer service when those events are unable to be directly written to a data store;

writing, via each events file writer of each event capture thread, the particular events file to a temporary events file storage system (TEFSS) that is configured to temporarily store one or more events files for subsequent writing to the data store;

executing, at one or more hardware-based processors of a second application server, an events file uploader service, wherein executing the events file uploader service comprises;

determining that the first application server is inactive;

reading at least one events file from the TEFSS that was written by the indirect events writer of the first application server; and

writing the plurality of events from each events file that was read to the data store.