Method and system for processing data for database modification

• US 9,086,927 B2
• Filed: 07/08/2011
• Issued: 07/21/2015
• Est. Priority Date: 06/28/2011
• Status: Active Grant

• Current Assignee: Amadeus S.A.S. (Amadeus Global Travel Distribution S.A.)
• Original Assignee: Amadeus S.A.S. (Amadeus Global Travel Distribution S.A.)
• Inventors: Becker, Muriel, Jullien, Ren, Moreau, Vincent
• Primary Examiner(s): Le, Miranda
• Application Number:
  US13/178,957
• Publication Number:
  US 20130006955A1
• Time in Patent Office: 1,474 Days
• Field of Search: 707/706, 718/104
• US Class Current: 1/1
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

A method and system for processing data for database modification, include receiving a set of data, performing a processing chain including a plurality of consecutive jobs to transform the set of data into transformed data, modifying a production database with respect to the transformed data and further include the steps of setting a target processing time for the performance of the consecutive jobs, before a launch of a first job, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs, before a launch of at least one further job after the first job, upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs remaining in the processing chain, the adapted configuration differing from the current configuration.

Claims

1. A method for processing data for database modification, the method comprising:

receiving a first set of data;

performing a processing chain comprising a plurality of consecutive jobs to transform the first set of data into transformed data;

modifying a database with respect to the transformed data;

setting a target processing time for performance of the consecutive jobs;

before the consecutive jobs are launched, applying a first configuration defining a first number of parallel instances for each of the consecutive jobs; and

if an actual remaining processing time for the consecutive jobs uncompleted in the processing chain is outside of a range of acceptable remaining processing times relative to the target processing time, applying a second configuration defining a second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain and including the first number of parallel instances for each of the consecutive jobs completed in the processing chain, the second number of parallel instances in the second configuration for at least one of the consecutive jobs uncompleted in the processing chain differing from the first number of parallel instances in the first configuration,

wherein the first configuration and the second configuration are determined based upon information comprising historical data and constraint data, and the historical data comprises information on previous executions of processing chains for sets of data of a same type as the first set of data.

2. The method of claim 1 wherein the range of acceptable remaining processing times is defined as a range of times under a determined proportion of the target processing time.

3. The method of claim 1 comprising:

creating a configuration table comprising, for several ranges of data volume, a configuration definition section including a plurality of configurations each specifying a number of parallel instances to be applied for each job,

wherein the first configuration is among the plurality of configurations.

4. The method of claim 3 further comprising:

storing the second configuration in the configuration table.

5. The method of claim 1 wherein the second configuration is applied if the actual remaining processing time is under the range of acceptable remaining processing times, and the second number of parallel instances to be applied to the remaining jobs is higher than the first number of parallel instances to be applied to the remaining jobs.

6. The method of claim 1 wherein the second configuration is applied if the actual remaining processing time is over the range of acceptable remaining processing times, and the second number of parallel instances to be applied to the remaining jobs is lower than the first number of parallel instances to be applied to the remaining jobs.

7. The method of claim 1, wherein the first set of data comprises data entities, each data entity describing a component of a fare definition of a travel product.

8. The method of claim 7 wherein the component is a fare, a fare rules, or a routing.

9. The method of claim 7 wherein the processing chain comprises:

a first job of conversion of the first set of data into a first set of data converted in a format of an internal structure,

a second job of construction of at least one preliminary table based on the converted first set of data,

a third job of update of a reference database based on the at least one preliminary table, and

a fourth job of load of the update of the reference database into a production database.

10. The method of claim 9 wherein the first job is using as input a file containing the first set of data.

11. The method of claim 9 wherein the fourth job is launched for at least one data entity for which the third job is completed even if the third job is not completed for all the data entities.

12. The method of claim 9 comprising:

receiving the first set of data for each of the data entities of components of fare definitions; and

performing parallel processing chains for said components the data entities.

13. The method of claim 1 comprising:

sending the first set of data from a provider system.

14. The method of claim 1 comprising:

accessing data of the production database from an end user device via a search engine.

15. The method of claim 1 further comprising:

upon receipt of a second set of data, detecting a dependent job of a processing chain of the second set of data that requires input data relying on results generated by a given job of the processing chain of the first set of data; and

scheduling a launch of the dependent job after the completion of the given job.

16. The method of claim 1 wherein applying the first configuration defining the first number of parallel instances for each of the consecutive jobs comprises:

selecting the first configuration based upon the data in the set numbering between a minimum and a maximum.

17. The method of claim 16 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is greater than the maximum.

18. The method of claim 16 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is less than the minimum.

19. A system comprising:

at least one data processor; and

program code configured upon execution by the at least one processor to process data for database modification by;

receiving a first set of data;

performing a processing chain comprising a plurality of consecutive jobs to transform the first set of data into transformed data;

modifying a database with respect to the transformed data;

setting a target processing time for performance of the consecutive jobs;

before the consecutive jobs are launched, applying a first configuration defining a first number of parallel instances for each of the consecutive jobs; and

if an actual remaining processing time for the consecutive jobs uncompleted in the processing chain is outside of a range of acceptable remaining processing times relative to the target processing time, applying a second configuration defining a second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain and including the first number of parallel instances for each of the consecutive jobs completed in the processing chain, the second number of parallel instances in the second configuration for at least one of the consecutive jobs uncompleted in the processing chain differing from the first number of parallel instances in the first configuration,

wherein the first configuration and the second configuration are determined based upon information comprising historical data and constraint data, and the historical data comprises information on previous executions of processing chains for sets of data of a same type as the first set of data.

20. The system of claim 19, wherein at least one of the first and second sets of data comprise data entities, each data entity describing a component of a fare definition of a travel product.

21. The system of claim 19 wherein the program code configured upon execution by the at least one processor to process data for database modification by:

upon receipt of a second set of data, detecting a dependent job of a processing chain of the second set of data that requires input data relying on results generated by a given job of the processing chain of the first set of data; and

scheduling a launch of the dependent job after the completion of the given job.

22. The system of claim 19 wherein the program code configured upon execution by the at least one processor to process data for database modification by applying the first configuration defining the first number of parallel instances for each of the consecutive jobs comprises:

program code configured upon execution by the at least one processor to process data for database modification by selecting the first configuration based upon the data in the set numbering between a minimum and a maximum.

23. The system of claim 22 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is greater than the maximum.

24. The system of claim 22 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is less than the minimum.

25. A computer program product comprising:

a non-transitory computer-readable storage medium; and

a computer program stored on the storage medium, the computer program comprising instructions that, when executed on a computer, cause the computer to process data for database modification by;

receiving a first set of data;

performing a processing chain comprising a plurality of consecutive jobs to transform the first set of data into transformed data;

modifying a database with respect to the transformed data;

setting a target processing time for performance of the consecutive jobs;

before the consecutive jobs are launched, applying a first configuration defining a first number of parallel instances for each of the consecutive jobs; and

if an actual remaining processing time for the consecutive jobs uncompleted in the processing chain is outside of a range of acceptable remaining processing times relative to the target processing time, applying a second configuration defining a second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain and including the first number of parallel instances for each of the consecutive jobs completed in the processing chain, the second number of parallel instances in the second configuration for at least one of the consecutive jobs uncompleted in the processing chain differing from the first number of parallel instances in the first configuration,

wherein the first configuration and the second configuration are determined based upon information comprising historical data and constraint data, and the historical data comprises information on previous executions of processing chains for sets of data of a same type as the first set of data.

26. The computer program product of claim 25, wherein at least one of the first and second sets of data comprise data entities, each data entity describing a component of a fare definition of a travel product.

27. The computer program product of claim 25 wherein the instructions that, when executed on the computer, further cause the computer to process data for database modification by:

upon receipt of a second set of data, detecting a dependent job of a processing chain of the second set of data that requires input data relying on results generated by a given job of the processing chain of the first set of data; and

scheduling a launch of the dependent job after the completion of the given job.

28. The computer program product of claim 25 wherein the instructions that, when executed on the computer, cause the computer to process data for database modification by applying the first configuration defining the first number of parallel instances for each of the consecutive jobs comprises:

instructions configured upon execution by the at least one processor to process data for database modification by selecting the first configuration based upon the data in the set numbering between a minimum and a maximum.

29. The computer program product of claim 28 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is greater than the maximum.

30. The computer program product of claim 28 wherein the second number of parallel instances for each of the consecutive jobs uncompleted in the processing chain is obtained from a third configuration that is applied if the data in the first set number is less than the minimum.