Method and system for importing data

• US 6,668,254 B2
• Filed: 12/21/2000
• Issued: 12/23/2003
• Est. Priority Date: 12/21/2000
• Status: Expired due to Fees

• Current Assignee: Fulltilt Asset Management Company
• Original Assignee: Full Tilt Solutions, Inc. (QAD Incorporated)
• Inventors: Clapper, Brian, Moyer, Bob, Stevens, Steve, Dymek, Matt, Matson, Ken, Hjellming, Tom
• Primary Examiner(s): Mizrahi, Diane D.
• Application Number:
  US09/741,766
• Publication Number:
  US 20020124005A1
• Time in Patent Office: 1,097 Days
• Field of Search: 707/10, 707/6, 707/2, 707/100, 707/101, 707/201, 707/523, 709/205, 709/201
• US Class Current: 1/1
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

A method and system for importing data comprising the downloading of product data from different sources and in different formats; processing the downloaded data by at least comparing it with data downloaded and stored in a product database; and reviewing the results of the comparison to detect differences in the data, the differences potentially being errors. The system and methods further comprise connecting the downloaded data from its supplier specific format into a standard format; comparing the downloaded data in the standard format with a previously downloaded data set saved in the standard format; categorizing the product data based on the results of the second comparison; and processing each category of data independently to automatically update the product database.

Claims

1. A data importation method comprising:

receiving first product data in a first format;

comparing the first product data with second product data previously received;

reviewing results of the comparison to determine whether there is a problem with the first product data;

changing a format of the first product data to a standard format;

comparing the standard format first product data with third product data, the third product data corresponding to the second product data having format changed to the standard format;

placing the standard format first product data in a category based on the comparison of the standard format first product data with the third product data; and

generating statistics based on the comparison of the standard format first product data with the third product data.

2. The method of claim 1, wherein placing the standard format first product data in a category comprises placing the standard format first product data in an identical products file.

3. The method of claim 1, wherein placing the standard format first product data in a category comprises placing the standard format first product data in a new products file.

4. The method of claim 3, further comprising:

retrieving original supplier data for an original supplier product;

normalizing at least one company in the retrieved supplier data;

looking up the original supplier product in a product database to determine whether data corresponding to the original supplier product has been provided by other suppliers;

locating a template for the original supplier product corresponding to the retrieved supplier data;

normalizing at least one attribute from the retrieved supplier data by using the template;

defining normalized product data as the supplier data having the normalized at least one company and the normalized at least one attribute; and

inserting the normalized product data into the product database.

5. The method of claim 4, wherein the step of normalizing at least one company comprises normalizing vendors and manufacturers associated with the product.

6. The method of claim 4, wherein the looking the product up step comprises determining whether the retrieved product data already exists in the product database.

7. The method of claim 6, further comprising:

comparing the normalized at least one attribute with existing attributes;

selecting correct attribute values; and

updating the normalized product data in the product database with the correct attribute values.

8. The method of claim 4, wherein looking up the retrieved product data step is performed by a human operator when an attempt to automatically perform the looking up step fails.

9. The method of claim 4, further comprising a step of assigning a template for the original supplier product data when a template has not been located automatically before normalizing the at least one attribute by using the template.

10. The method of claim 4, further comprising a step of updating attribution definitions before the step of inserting the normalized product data.

11. The method of claim 4, further comprising:

identifying a category associated with the original supplier product;

retrieving original supplier data for other original supplier products; and

optionally assigning to the located template all products in the other supplier original products corresponding to the identified category.

12. The method of claim 4, further comprising:

retrieving original supplier data for other original supplier products similar to manually-assigned product; and

optionally assigning to the located template original supplier products in the original supplier data that are similar to the manually-assigned product.

13. The method of claim 1, wherein placing the standard format first product data in a category comprises placing the standard format first product data in a changed products file.

14. The method of claim 1, wherein placing the standard format first product data in a category comprises placing the standard format first product data in a deleted products file.

15. The method of claim 14, further comprising:

retrieving product data from the delete products file;

looking up the retrieved product data in the product database;

deleting from the database the retrieved product data, which corresponds to a first supplier, when a product corresponding to the retrieved product data has not been deleted for all other suppliers.

16. The method of claim 15, further comprising:

marking the product as deleted when the first supplier is an only supplier having the product undeleted in the database at the time of looking up the retrieved product data in the database.

17. The method of claim 1, wherein placing the standard format first product data in a category comprises placing the standard format first product data in a faulty products file.

18. A method for normalizing product data comprising:

retrieving original supplier data for an original supplier product;

normalizing at least one company in the retrieved supplier data;

looking up the original supplier product in a product database to determine whether data corresponding to the original supplier product has been provided by other suppliers;

locating a template for the original supplier product corresponding to the retrieved supplier data;

normalizing at least one attribute from the retrieved supplier data by using the template;

defining normalized product data as the supplier data having the normalized at least one company and the normalized at least one attribute; and

inserting the normalized product data into the product database.

19. The method of claim 18, wherein the step of normalizing at least one company comprises normalizing vendors and manufacturers associated with the product.

20. The method of claim 18, wherein the looking the product up step comprises determining whether the retrieved product data already exists in the product database.

21. The method of claim 20, further comprising:

comparing the normalized at least one attribute with existing attributes;

selecting correct attribute values; and

updating the normalized product data in the product database with the correct attribute values.

22. The method of claim 18, wherein looking up the retrieved product data step is performed by a human operator when an attempt to automatically perform the looking up step fails.

23. The method of claim 18, further comprising a step of assigning a template for the original supplier product data when a template has not been located automatically before normalizing the at least one attribute by using the template.

24. The method of claim 18, further comprising a step of updating attribution definitions before the step of inserting the normalized product data.

25. The method of claim 18, further comprising:

identifying a category associated with the original supplier product;

retrieving original supplier data for other original supplier products; and

optionally assigning to the located template all products in the other supplier original products corresponding to the identified category.

26. The method of claim 18, further comprising:

retrieving original supplier data for other original supplier products similar to manually-assigned product; and

optionally assigning to the located template original supplier products in the original supplier data that are similar to the manually-assigned product.

27. A method for processing data comprising:

receiving data from a first queue;

applying a data transformation rule to process the received data;

transferring the processed data to a second queue if the application of the data transformation rule is successful; and

performing the following steps if the application of the data transformation rule is not successful;

transferring the received data to a third queue;

receiving manual commands for processing the data in the third queue;

processing the data in the third queue according to the manual commands; and

transferring to a fourth queue the data processed from the third queue.

28. The method of claim 27, wherein the application of the data transformation rule is not successful when the data received from the first queue is malformed.

29. The method of claim 27, wherein the application of the data transformation rule is not successful when the data received from the first queue is of an unknown type.

30. The method of claim 27, wherein the application of the data transformation rule is not successful when the data received from the first queue does not meet data cleansing definitions.

31. The method of claim 27, wherein the step of receiving manual commands comprises receiving a new data transformation rule.

32. The method of claim 27, wherein the step of receiving manual commands comprises receiving changes in the received data.

33. The method of claim 27, wherein the step of receiving manual commands comprises receiving instructions to discard the received data.

34. The method of claim 27, further comprising a step of saving the data transformation rule for use in a next import from the first queue.

35. The method of claim 27, wherein the step of applying a data transformation rule comprises normalizing product attribute data received from the first queue.

36. The method of claim 35, wherein the step of receiving manual commands comprises receiving updated attribution definitions.

37. The method of claim 27, wherein the step of receiving data from a first queue comprises receiving product data.

38. The method of claim 27, wherein the step of applying a data transformation rule comprises normalizing product manufacturer data received from the first queue.

39. The method of claim 38, the step of receiving manual commands comprises receiving a new company name.

40. The method of claim 38, further comprising:

searching for the product manufacturer in a database; and

determining that the step of applying the data transformation rule is successful when the product manufacturer is found in the database.

41. The method of claim 38, further comprising:

searching for the data transformation rule in a database; and

determining that the step of applying the data transformation rule is successful when the rule is found in the database.

42. The method of claim 38, further comprising:

searching for the product manufacturer in a first database;

searching for the data transformation rule in a second database; and

determining that the step of applying the data transformation rule is not successful when the product manufacturer is not found in the first database and the data transformation rule is not found in the second database.

43. The method of claim 42, wherein the first and second databases are the same.

44. The method of claim 27, wherein the step of applying a data transformation rule comprises normalizing product category data received from the first queue.

45. The method of claim 44, wherein the step of receiving manual commands comprises receiving a new product category.

46. The method of claim 44, further comprising:

searching for the product category in a database; and

determining that the step of applying the data transformation rule is not successful when the product category is not found in the database.

47. The method of claim 44, further comprising:

searching for the data transformation rule in a database; and

determining that the step of applying the data transformation rule is not successful when the rule is not found in the database.

48. The method of claim 44, wherein the step of normalizing product category data comprises assigning to a product a template corresponding to a product category.

49. The method of claim 27, wherein the fourth queue is the second queue.

50. The method of claim 27, wherein the fourth queue is the first queue.

51. The method of claim 27, wherein the step of receiving manual commands comprises receiving changes to the data transformation rule.

52. The method of claim 35, further comprising:

searching for a template attribute in a database; and

creating a product attribute corresponding to said template attribute.

53. In a data processing system, a computer-readable medium providing instructions that when implemented by a computer perform a method comprising:

receiving data from a first queue;

applying a data transformation rule to process the received data;

transferring the processed data to a second queue if the application of the data transformation rule is successful; and

performing the following steps if the application of the data transformation rule is not successful;

transferring the received data to a third queue;

receiving manual commands for processing the data in the third queue;

processing the data in the third queue according to the manual commands; and

transferring to a fourth queue the data processed from the third queue.

54. The computer-readable medium of claim 53, wherein the fourth queue is the second queue.

55. The computer-readable medium of claim 53, wherein the fourth queue is the first queue.