HEADER-TOKEN DRIVEN AUTOMATIC TEXT SEGMENTATION

• US 20150261761A1
• Filed: 05/28/2015
• Published: 09/17/2015
• Est. Priority Date: 12/28/2006
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: eBay Inc.
• Inventors: Sarwar, Badrul M., Mount, John A.
• Granted Patent: US 9,529,862 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

A method and a system to automatically segment text based on header tokens is described. A relevance value and an irrelevance value are determined for each token in a description, assuming no tokens are left out of computations. The irrelevance value is based on occurrences of a token in a sample set of descriptions. The relevance value is an estimated probability of relevance based on the header of the description being segmented.

Claims

1. (canceled)

3. The system of claim 1, wherein:

the assigned value further indicates a relevance probability of the first token; and

the computing of the relevance value of the group of tokens is based on the relevance probability of the first token and further based on an irrelevance probability of the first token.

4. The system of claim 3, the processor-implemented segmentation module further configured to determine the irrelevance probability based on a proportion of descriptions that contain the first token within a set of descriptions.

5. The system of claim 1, the processor-implemented segmentation module further configured to determining that the lexical association exists between the first token and the second token by accessing a table that lexically associates the first token with the second token.

6. The system of claim 1, wherein:

the computing of the relevance value of the group of tokens computes the relevance value of a segment of the description; and

the indicating indicates that the segment is a most relevant segment of the description.

7. The system of claim 1, wherein:

the computing of the relevance value of the group of tokens computes the relevance value of a sequence of tokens; and

the indicating indicates that the sequence is a most relevant sequence of tokens in the description.

8. The system of claim 1, wherein the indicating that the group of tokens is the most relevant group of tokens in the description includes modifying a font of the most relevant group of tokens in the description.

9. The system of claim 1, wherein the indicating that the group of tokens is the most relevant group of tokens in the description includes presenting the most relevant group of tokens with the title of the description.

2. A system comprising:

a processor-implemented segmentation module configured to;

receive data from a client machine, the data comprising a product title and a product description;

identify a first token in the product description;

receive a token probability value associated with the first token;

assign a value to the first token, the value indicating that, one of;

the first token also occurs in the product title,

a lexical association exists between the first token and a second token in the product title, and

the lexical association does not exist and the first token is absent from the product title;

compute a relevance value of a group of tokens that occur in the product description and include the first token with the assigned value, the relevance value of the group computed based on the value assigned to the first token; and

determine and store in memory an indication that the group of tokens is a most relevant group of tokens in the product description;

wherein the assigning of the value to the first token includes;

initially assigning and storing a default value that indicates the lexical association does not exist and the first token is absent from the product title; and

overwriting the stored initially assigned default value based on the first token occurring in the product title.

10. A method implemented on a processor-implemented segmentation module, the method comprising:

receiving data from a client machine, the data comprising a product title and a product description;

identifying a first token in the product description;

receiving a token probability value associated with the first token;

assigning a value to the first token, the value indicating that, one of;

the first token also occurs in the product title,

a lexical association exists between the first token and a second token in the product title, and

the lexical association does not exist and the first token is absent from the product title;

computing a relevance value of a group of tokens that occur in the product description and include the first token with the assigned value, the relevance value of the group computed based on the value assigned to the first token; and

determining and store in memory an indication that the group of tokens is a most relevant group of tokens in the product description;

wherein the assigning of the value to the first token includes;

initially assigning and storing a default value that indicates the lexical association does not exist and the first token is absent from the product title; and

overwriting the stored initially assigned default value based on the first token occurring in the product title.

11. The method of claim 10, wherein:

the assigned value further indicates a relevance probability of the first token; and

the computing of the relevance value of the group of tokens is based on the relevance probability of the first token and further based on an irrelevance probability of the first token.

12. The method of claim 11, the method further comprising determining the irrelevance probability based on a proportion of descriptions that contain the first token within a set of descriptions.

13. The method of claim 10, the method further comprising determining that the lexical association exists between the first token and the second token by accessing a table that lexically associates the first token with the second token.

14. The method of claim 10, wherein:

the computing of the relevance value of the group of tokens computes the relevance value of a segment of the description; and

the indicating indicates that the segment is a most relevant segment of the description.

15. The method of claim 10, wherein:

the computing of the relevance value of the group of tokens computes the relevance value of a sequence of tokens; and

the indicating indicates that the sequence is a most relevant sequence of tokens in the description.

16. The method of claim 10, wherein the indicating that the group of tokens is the most relevant group of tokens in the description includes modifying a font of the most relevant group of tokens in the description.

17. The method of claim 10, wherein the indicating that the group of tokens is the most relevant group of tokens in the description includes presenting the most relevant group of tokens with the title of the description.

18. A system comprising:

a processor-implemented segmentation module configured to;

receive data from a client machine, the data comprising a product header and a product description;

identify a first token in the product description;

receive a token probability value associated with the first token;

assign a value to the first token, the value indicating that, one of;

the first token also occurs in the product header,

a lexical association exists between the first token and a second token in the product header, and

the lexical association does not exist and the first token is absent from the product header;

compute a relevance value of a group of tokens that occur in the product description and include the first token with the assigned value, the relevance value of the group computed based on the value assigned to the first token; and

determining and storing in memory an indication that the group of tokens is a most relevant group of tokens in the product description;

wherein the assigning of the value to the first token includes;

initially assigning and storing a default value that indicates the lexical association does not exist and the first token is absent from the product header; and

overwriting the stored initially assigned default value based on the first token occurring in the product header.

19. The system of claim 18, wherein:

the header of the description is a body of text shorter than the description and provides information about the description.

20. The system of claim 18, wherein:

the header of the description includes at least one of a title of the description, an abstract of the description, a summary of description, or a synopsis of the description.

21. The system of claim 18, the processor-implemented segmentation module further configured to determine an irrelevance probability of the first token based on a proportion of descriptions that contain the first token within a set of descriptions, wherein:

the assigned value further indicates a relevance probability of the first token; and

the computing of the relevance value of the group of tokens is based on the relevance probability of the first token and further based on the irrelevance probability.