METHOD AND SYSTEM FOR PROCESSING DATA FOR DATABASE MODIFICATION

• US 20130006955A1
• Filed: 07/08/2011
• Published: 01/03/2013
• Est. Priority Date: 06/28/2011
• Status: Active Grant

• Current Assignee: Amadeus (Amadeus Global Travel Distribution S.A.)
• Original Assignee: Amadeus (Amadeus Global Travel Distribution S.A.)
• Inventors: JULLIEN, René, MOREAU, Vincent, BECKER, Muriel
• Granted Patent: US 9,086,927 B2
• Time in Patent Office: Days
• US Class Current: 707/706
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

A method and system for processing data for database modification, include receiving a set of data, performing a processing chain including a plurality of consecutive jobs to transform the set of data into transformed data, modifying a production database with respect to the transformed data and further include the steps of setting a target processing time for the performance of the consecutive jobs, before a launch of a first job, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs, before a launch of at least one further job after the first job, upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs remaining in the processing chain, the adapted configuration differing from the current configuration.

Claims

1. Method for processing data for database modification, comprising the steps of:

receiving a first set of data,

performing a processing chain comprising a plurality of consecutive jobs to transform the first set of data into transformed data,

modifying a production database with respect to the transformed data,

setting a target processing time for the performance of the consecutive jobs,

before a launch of a first job of the consecutive jobs, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs,

upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs, of the consecutive jobs, remaining in the processing chain, said adapted configuration differing from the current configuration, and

upon receipt of at least a second set of data, performing a step of detecting a dependent job and scheduling a launch of the dependent job, including

i) detecting the dependent job of a processing chain of the second set of data which requires an input data relying on results generated by at least a given job of the processing chain of the first set of data, and

ii) scheduling a launch of said dependent job after the completion of said given job.

2. The method of claim 1 further comprising:

before a launch of all further jobs after the first job, upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs remaining in the processing chain, said adapted configuration differing from the current configuration.

3. The method according to claim 1 wherein the range of acceptable remaining processing times is defined as a range of times under a determined proportion of the target processing time.

4. The method according to claim 1 wherein the original configuration and the adapted configuration are determined on the basis of information comprising historical data and constraint data.

5. The method of claim 4 wherein the historical data comprise information on previous executions of processing chains for sets of data of a same type as the first set of data.

6. The method of claim 5 comprising the step of creating a configuration table comprising, for several ranges of data volume, a configuration definition section including at least one configuration definition specifying a number of parallel instances to be applied for each job.

7. The method of claim 6 comprising, upon the actual remaining processing time being under the range of acceptable remaining processing times, choosing the adapted processing configuration among at least one configuration of the configuration table for which the number of parallel instances to be applied to the remaining jobs is higher than the number of instances to be applied to the remaining jobs of the current configuration.

8. The method of claim 6 comprising, upon the actual remaining processing time being over the range of acceptable remaining processing times, choosing the adapted processing configuration among at least one configuration of the configuration table for which the number of parallel instances to be applied to the remaining jobs is lower than the number of instances to be applied to the remaining jobs of the current configuration.

10. The method of claim 1, wherein at least one of the first and second sets of data comprise data entities each describing one component of a fare definition of a travel product.

11. The method of claim 10 wherein the component is selected among fare and fare rules and routings.

12. The method according to claim 10 wherein the processing chain comprises:

a first job of conversion of the first set of data into a first set of data converted in a format of an internal structure,

a second job of construction of at least one preliminary table based on the converted first set of data,

a third job of update of a reference database based on the at least one preliminary table, and

a fourth job of load of the update of the reference database into a production database.

13. The method of claim 12 wherein the first job is using as input a file containing the first set of data.

14. The method of claim 12 wherein the fourth job is launched for at least one data entity for which the third job is completed even if the third job is not completed for all the data entities.

15. The method of method of claim 12 comprising receiving the first set of data for each of components of fare definitions and performing parallel processing chains for said components.

16. The method of claim 1 performed as a result of execution of computer software by at the least one data processor, the computer software being stored in a non-transitory computer-readable memory medium.

17. The method of claim 1 comprising sending the first set of data from a provider system.

18. The method of claim 1 comprising accessing data of the production database from and end user device via a search engine.

9. (canceled)

19. (canceled)

20. System for processing data for database modification, the system comprising:

at least one data processor; and

a non-transitory computer-readable memory medium storing computer software that, when executed by the at least one data processor, causes the at least one data processor to execute

a) setting a target processing time for the performance of consecutive jobs, including i) before a launch of a first job, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs, and ii) upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs remaining in the processing chain, said adapted configuration differing from the current configuration; and

b) a job scheduler, wherein, upon receipt of at least a second set of data, detecting a dependent job of a processing chain of the second set of data which requires an input data relying on results generated by at least a given job of the processing chain of the first set of data, and scheduling a launch of said dependent job after the completion of said given job.

25. The system of claim 20, wherein at least one of the first and second sets of data comprise data entities, each data entity describing one component of a fare definition of a travel product.

21-23. -23. (canceled)

24. A non-transitory computer-readable memory medium storing a computer program comprising instructions that when executed on a computer causes the computer to process data for database modification, by executing the steps of:

receiving a first set of data;

performing a processing chain comprising a plurality of consecutive jobs to transform the first set of data into transformed data;

modifying a production database with respect to the transformed data;

setting a target processing time for the performance of the consecutive jobs;

before a launch of a first job of the consecutive jobs, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs;

upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs, of the consecutive jobs, remaining in the processing chain, said adapted configuration differing from the current configuration; and

upon receipt of at least a second set of data, performing a step of detecting a dependent job and scheduling a launch of the dependent job, including

i) detecting the dependent job of a processing chain of the second set of data which requires an input data relying on results generated by at least a given job of the processing chain of the first set of data, and

ii) scheduling a launch of said dependent job after the completion of said given job.

26. The non-transitory computer-readable memory medium of claim 24, wherein at least one of the first and second sets of data comprise data entities, each data entity describing one component of a fare definition of a travel product.