Method for automatic categorization of items

• US 6,751,600 B1
• Filed: 05/30/2000
• Issued: 06/15/2004
• Est. Priority Date: 05/30/2000
• Status: Expired due to Term

• Current Assignee: Perfect Commerce Holdings, LLC
• Original Assignee: Commerce One Operations Incorporated (Proactis Holdings Plc)
• Inventors: Wolin, Ben
• Primary Examiner(s): Khatri, Anil
• Assistant Examiner(s): Hirl, Joseph P.
• Application Number:
  US09/583,650
• Time in Patent Office: 1,477 Days
• Field of Search: 706/12, 706/59, 706/902, 707/100
• US Class Current: 706/12
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: Y10S 706/902 Application using ai with detail of the ai system

Abstract

A system and method for automatic categorization of items into categories. Machine learning establishes or updates a data structure including term weights for text fields and distributions for numeric fields, based on a sample of pre-categorized items. An automatic categorization engine processes items by referencing the data structure on a field-by-field basis, determining a ranking score for each alternative category to which an item may be assigned. A category assignment may be based on ranking scores and may be flagged for a user to review. A user interface facilitates review and confirmation of automatic category assignments, either comprehensively, as flagged by the automatic categorization engine, or according to user determined criteria.

Claims

1. A method of machine learning to automatically categorize items from a plurality of pre-categorized items having fields, wherein the fields include one or more text fields having terms and one or numeric fields having values, including the steps:

(a) counting a frequency of usage of terms by category for one or more text fields in a plurality of pre-categorized items, wherein a term is a single word or both single words and phrases;

(b) weighting the frequency of usage of a particular term in a particular category based on the frequency of usage of said particular term in other categories; and

(c) determining a distribution by category for values in one or more numeric fields of said pre-categorized items.

2. The method of claim 1, wherein at least one value in a numeric field is a price for the pre-categorized item.

3. The method of claim 1, wherein at least one value in a numeric field is a dimension for the pre-categorized item.

4. The method of claim 1, wherein an item is a product to be listed in a product catalog.

5. The method of claim 1, wherein a category is a product category in a product catalog.

6. The method of claim 1, further including the step of storing the weighting of frequency usage and the distribution in a sparse matrix.

7. The method of claim 1, further including the step of storing the weighting of frequency usage and the distribution in a B-tree.

8. The method of claim 1, further including the step of storing the weighting of frequency usage and the distribution in a database.

9. The method of claim 1, wherein the weighting is determined by a term frequency—

inverse document frequency ranking algorithm.

10. The method of claim 1, further including the step of filtering said pre-categorized items by values in one or more numeric fields to eliminate outliers,

wherein the determining step precedes the filtering step which precedes the counting step.

11. The method of claim 10, wherein the filtering step eliminates outliers that are more than a predetermined number of standard deviations from the mean value for a numeric field.

12. The method of claim 10, wherein the filtering step eliminates predetermined percentiles of highest and lowest outliers.

13. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

further including the step of normalizing ranking cores based on the number of parsed terms.

14. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

further including the step of selecting one or more categories based on the adjusted ranking scores.

15. The method of claim 14, further including the step of determining whether to flag the uncategorized item for human review.

16. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

further including the step of rank ordering categories based on the adjusted ranking scores.

17. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

wherein calculating ranking scores for an identified category includes

summing the weighted frequencies for the parsed terms;

normalizing the sum of the weighted frequencies based on the number of parsed terms in the uncategorized item.

18. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

wherein calculating ranking scores for an identified category includes

summing by text field the weighted frequencies for the parsed terms;

combining the sums of weighted frequencies by text field according to a predetermined weighting formula;

normalizing the combined sum of weighted frequencies.

19. The method of claim 18, wherein the predetermined weighting formula assigns a greater weight to a text field containing a short description of the uncategorized item than a text field containing a long description of the uncategorized item.

20. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

wherein the adjusting step includes applying a multiplicative factor to said ranking scores.

21. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

wherein the adjusting step includes applying an additive factor to said ranking scores.

22. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying categories associated with the terms;

(c) calculating ranking scores for the terms in the identified categories; and

(d) adjusting said ranking scores corresponding to a comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

wherein the adjusting step includes applying a decision rule to said ranking scores.

23. A method of automatically categorizing an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) determining for each text field the applicable ranking ideas;

(c) identifying categories associated with the terms for the applicable ranking ideas;

(d) calculating ranking scores for the terms for the applicable ranking ideas in the identified categories; and

(e) adjusting said ranking scores corresponding to comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

further including the step of selecting one or more categories based on the adjusted ranking scores.

24. The method of claim 23, further including the step of normalizing ranking scores based on the number of parsed terms.

25. The method of claim 23, further including the step of determining whether to flag the uncategorized item for human review along with adjusted ranking scores for a plurality of the identified categories.

26. The method of claim 23, wherein calculating ranking scores for an identified category includes

summing weighted frequencies for the parsed terms;

normalizing the sum of the weighted frequencies based on the number of parsed terms in the uncategorized item.

27. The method of claim 23, wherein calculating ranking scores for an identified category includes

summing weighted frequencies for the parsed terms;

combining the sums of weighted frequencies according to a predetermined weighting formula;

normalizing the combined sum of weighted frequencies.

28. The method of claim 23, wherein the predetermined weighting formula assigns a greater weight to a ranking field containing a short description of the uncategorized item than a text field containing a long description of the uncategorized item.

29. The method of claim 23, wherein the adjusting step includes applying a multiplicative factor to said ranking scores.

30. The method of claim 23, wherein the adjusting step includes applying an additive factor to said ranking scores.

31. The method of claim 23, wherein the adjusting step includes applying a decision rule to said ranking scores.

32. A method of automatically categorizing an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, and category by category data is available for a frequency of term usage and a distribution of values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) determining for each text field the applicable ranking ideas;

(c) identifying categories associated with the terms for the applicable ranking ideas;

(d) calculating ranking scores for the terms for the applicable ranking ideas in the identified categories; and

(e) adjusting said ranking scores corresponding to comparison of values in one or more numeric fields of the uncategorized item with corresponding distributions of numeric values;

further including the step of rank ordering categories based on the adjusted ranking scores.

33. A method of machine learning to automatically categorize items from a plurality of pre-categorized items having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) counting a frequency of usage of terms in one or more text fields in a plurality of pre-categorized items wherein a term is a single word or both single words and phrases;

(b) weighting the frequency of usage of a particular term in a particular item based on the frequency of usage of said particular term in other items; and

(c) determining a distribution by category for values in one or more numeric fields of said pre-categorized items.

34. The method of claim 33, wherein at least one value in a numeric field is a price for the pre-categorized item.

35. The method of claim 33, wherein at least one value in a numeric field is a dimension for the pre-categorized item.

36. The method of claim 33, wherein an item is a product to be listed in a product catalog.

37. The method of claim 33, wherein a category is a product category in a product catalog.

38. The method of claim 33, further including the step of storing the weighting of frequency usage and the distribution in a sparse matrix.

39. The method of claim 33, further including the step of storing the weighting of frequency usage and the distribution in a B-tree.

40. The method of claim 33, further including the step of storing the weighting of frequency usage and the distribution in a database.

41. The method of claim 33, wherein the weighting is determined by a term frequency—

inverse document frequency ranking algorithm.

42. The method of claim 33, further including the step of filtering said pre-categorized items by values in one or more numeric fields to eliminate outliers,

wherein the determining step precedes the filtering step which precedes the counting step.

43. The method of claim 42, wherein the filtering step eliminates outliers that are more than a predetermined number of standard deviations from the mean value for a numeric field.

44. The method of claim 42, wherein the filtering step eliminates predetermined percentiles of highest and lowest outliers.

45. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item

wherein the adjusting step further includes comparing the values in one or more numeric fields of the uncategorized item with corresponding values in numeric fields of the identified items.

46. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein category by category data is available for one or more distributions of values and the adjusting step compares the values in one or more numeric fields of the uncategorized item with the distributions of values for categories corresponding to the identified items.

47. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

further including the step of normalizing ranking scores based on the number of parsed terms.

48. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

further including the step of selecting one or more categories based on the adjusted ranking scores.

49. The method of claim 48, further including the step of determining whether to flag the uncategorized item for human review.

50. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

further including the step of rank ordering categories based on the adjusted ranking scores.

51. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein calculating ranking scores for an identified category includes

summing the weighted frequencies for the parsed terms;

normalizing the sum of the weighted frequencies based on the number of parsed terms in the uncategorized item.

52. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein calculating ranking scores for an identified category includes

summing by text field the weighted frequencies for the parsed terms;

combining the sums of weighted frequencies by text field according to a predetermined weighting formula;

normalizing the combined sum of weighted frequencies.

53. The method of claim 52, wherein the predetermined weighting formula assigns a greater weight to a text field containing a short description of the uncategorized item than a text field containing a long description of the uncategorized item.

54. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein the adjusting step includes applying a multiplicative factor to said ranking scores.

55. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein the adjusting step includes applying an additive factor to said ranking scores.

56. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or more numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) identifying items associated with the terms;

(c) calculating ranking scores for the terms in the identified items; and

(d) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein the adjusting step includes applying a decision rule to said ranking scores.

57. A method of ranking for automatic categorization of an item having fields, wherein the fields include one or more text fields having terms and one or numeric fields having values, including the steps:

(a) parsing terms in one or more text fields of an uncategorized item;

(b) determining for each text field the applicable categorization ideas;

(c) identifying items associated with the terms for the applicable categorization ideas;

(d) calculating ranking scores for the terms for the applicable categorization ideas in the identified items; and

(e) adjusting said ranking scores based on values in one or more numeric fields of the uncategorized item;

wherein the adjusting step further includes comparing the values in one or more numeric fields of the uncategorized item with corresponding values in numeric fields of the identified items.

58. The method of claim 57, wherein category by category data is available for one or more distributions of values and the adjusting step compares the values in one or more numeric fields of the uncategorized item with the distributions of values for categories corresponding to the identified items.

59. The method of claim 57, further including the step normalizing ranking scores based on the number of parsed terms.

60. The method of claim 57, further including the step of selecting one or more categories based on the adjusted ranking scores.

61. The method of claim 60, further including the step of determining whether to flag the uncategorized item for human review along with adjusted ranking scores for a plurality of the identified categories.

62. The method of claim 57, further including the step of rank ordering categories based on the adjusted ranking scores.

63. The method of claim 57, wherein calculating ranking scores for an identified category includes

summing weighted frequencies for the parsed terms;

normalizing the sum of the weighted frequencies based on the number of parsed terms in the uncategorized item.

64. The method of claim 57, wherein calculating ranking scores for an identified category includes

summing weighted frequencies for the parsed terms;

combining the sums of weighted frequencies according to a predetermined weighting formula;

normalizing the combined sum of weighted frequencies.

65. The method of claim 64, wherein the predetermined weighting formula assigns a greater weight to a ranking field containing a short description of the uncategorized item than a text field containing a long description of the uncategorized item.

66. The method of claim 57, wherein the adjusting step includes applying a multiplicative factor to said ranking scores.

67. The method of claim 57, wherein the adjusting step includes applying an additive factor to said ranking scores.

68. The method of claim 57, wherein the adjusting step includes applying a decision rule to said ranking scores.