Method and system for data file processing

• US 7,788,638 B2
• Filed: 11/21/2003
• Issued: 08/31/2010
• Est. Priority Date: 11/21/2003
• Status: Active Grant

• Current Assignee: Citigroup Global Markets Incorporated (Citigroup Incorporated)
• Original Assignee: Citigroup Global Markets Incorporated (Citigroup Incorporated)
• Inventors: Hoberman, Gary A., D'Souza, Lestan
• Primary Examiner(s): CHAVIS, JOHN Q
• Application Number:
  US10/718,885
• Publication Number:
  US 20050114287A1
• Time in Patent Office: 2,475 Days
• Field of Search: 717/120
• US Class Current: 717/120
• CPC Class Codes: G06F 8/20 Software design

Abstract

A library of jobs is created, with the jobs performing predetermined functions. The jobs include a termination indication, which is success or failure. Input file formats of known file types are determined and a job stream to process the input files is constructed of the jobs linked according to the termination success or failure of the preceding job.

Claims

1. A method implemented at least partially in a programmed computer for software application development suitable for processing input files with varying file formats or data structures, the method comprising:

characterizing the file format and data structure of at least one known input file type using the programmed computer; and

creating a library comprising a plurality of jobs to perform predetermined functions using the programmed computer, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure, and the plurality of jobs can be linked according to the indicator of job termination success or job termination failure.

2. A method according to claim 1, further comprising:

creating a job stream library using the programmed computer, the job stream library comprising at least one job stream, the job stream comprising individual jobs from the library of jobs, and the at least one job stream in the job stream library including an indicator of job stream termination, wherein the indicator is either job stream termination success or job stream termination failure.

3. A method according to claim 1, further comprising adding jobs to the library of jobs using the programmed computer.

4. A system implemented at least partially in a programmed computer for software application development suitable for processing input files with varying file formats or data structures, the system comprising a processor with:

computer program means for characterizing the file format and data structure of at least one known input file type; and

computer program means for creating a library comprising a plurality of jobs to perform predetermined functions, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure, and the plurality of jobs can be linked according to the indicator of job termination success or job termination failure.

5. A computer-executable program that is tangibly embodied on a computer-readable medium as computer executable software code, the code for software application development suitable for processing input files with varying file formats or data structures, the code comprising:

code to characterize the file format and data structure of at least one known input file type; and

code to create a library comprising a plurality of jobs to perform predetermined functions, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure, and the plurality of jobs can be linked according to the indicator of job termination success or job termination failure.

6. A programmed computer for software application development suitable for processing input files with varying file formats or data structures, comprising:

a memory having at least one region for storing computer executable program code; and

a processor for executing the program code stored in the memory, wherein the program code comprises;

code to characterize the file format and data structure of at least one known input file type; and

code to create a library comprising a plurality of jobs to perform predetermined functions, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure, and the plurality of jobs can be linked according to the indicator of job termination success or job termination failure.

7. A system implemented at least partially in a programmed computer for software application development, comprising a processor with:

computer program means for selecting a plurality of jobs from a preexisting library of jobs, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure;

computer program means for creating a job stream, the job stream comprising the plurality of jobs linked to each other according to the indicator of job termination success or job termination failure;

computer program means for creating a manager;

computer program means for receiving the data file; and

computer program means for processing the data file with the job stream in a plurality of steps, either in sequence or in parallel, using the manager, wherein at least one of the plurality of steps comprises converting the file format and data structure of the data file to the tab delimited file format or data structure common to the plurality of jobs.

8. A computer-executable program that is tangibly embodied on a computer-readable medium as computer executable software code, the code for software application development, the code comprising:

code to select a plurality of jobs from a preexisting library of jobs, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure;

code to create a job stream, the job stream comprising the plurality of jobs linked to each other according to the indicator of job termination success or job termination failure;

code to create a manager;

code to receive the data file; and

code to process the data file with the job stream in a plurality of steps, either in sequence or in parallel, using the manager, wherein at least one of the plurality of steps comprises converting the file format and data structure of the data file to the tab delimited file format or data structure common to the plurality of jobs.

9. A programmed computer for software application development, comprising:

a memory having at least one region for storing computer executable program code; and

a processor for executing the program code stored in the memory, wherein the program code comprises;

code to select a plurality of jobs from a preexisting library of jobs, wherein the library of jobs comprises at least one job to read the file format and data structure of the known input file and convert that input file to a tab delimited file format or data structure that is common to the plurality of jobs, at least one job for extracting data from a file, at least one job for archiving files, at least one job for loading files, at least one job for messaging, at least one job for transforming data in a file, at least one job for validating data in a file, and further wherein each job includes an indicator of job termination success or job termination failure;

code to create a job stream, the job stream comprising the plurality of jobs linked to each other according to the indicator of job termination success or job termination failure;

code to create a manager;

code to receive the data file; and

code to process the data file with the job stream in a plurality of steps, either in sequence or in parallel, using the manager, wherein at least one of the plurality of steps comprises converting the file format and data structure of the data file to the tab delimited file format or data structure common to the plurality of jobs.