Construction of a lexicon for a selected context

• US 9,697,195 B2
• Filed: 05/19/2015
• Issued: 07/04/2017
• Est. Priority Date: 10/15/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Zhen, Chuang, Darren, Li, Jingmei, Bobby, Mak Chiu Chun
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US14/716,244
• Publication Number:
  US 20160110341A1
• Time in Patent Office: 777 Days
• CPC Class Codes: G06F 40/129 Handling non-Latin characters, e.g. kana-to-kanji conversion
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 5/01 for shifting, e.g. justifying, scaling, normalising digital stores in which the information is moved stepwise, e.g. shift-registers G11C19/00; digital stores in which the information circulates G11C21/00

Abstract

Various technologies pertaining to constructing a lexicon for a defined context are set forth herein. Social media text is acquired, where the social media text has contextual data that corresponds thereto. The social media text is encoded to form encoded text (in Unicode), and the contextual data is assigned to the encoded text. A text corpus for a defined context is formed by filtering the encoded text based upon contextual data, such as location. Frequency of occurrence of words or phrases in the text corpus is used to identify words or phrases that are to be included in the lexicon.

Claims

1. A computing system that generates a lexicon of a social network, the computing system comprising:

a processor; and

memory storing instructions that, when executed by the processor, provide;

a lexicon generator that;

for respective messages of the social network;

identifies a context of the message,

scans the message to identify a set of word sequences, and

counts an occurrence of the respective word sequences among the messages within the context;

for respective contexts, generates a lexicon for the context that comprises the word sequences identified in the messages with a higher count than a word sequence count threshold;

identifies a selected word sequence by which at least one user of the social network who is associated with the selected word sequence is identifiable; and

refrains from including the selected word sequence from the lexicon; and

a lexicon presenter that, responsive to a selection by a user of a selected context, presents to the user the lexicon of word sequences for the selected context.

2. The computing system of claim 1, wherein a word sequence of selected message further comprises a first word in a first language and a second word in a second language.

3. The computing system of claim 1, wherein the contexts of respective messages are based upon at least one of:

a gender of generators or sharers of the message;

a geographic location of the generators or sharers of the message;

an age range of the generators or sharers of the message; and

explicitly defined interests of the generators or sharers of the message.

4. The computing system of claim 1, wherein the lexicon generator scans the respective messages to identify the word sequences by:

incrementally selecting a sequence length; and

partitioning the message into word sequences of the sequence length that are to be counted for occurrence among the messages by the scan.

5. The computing system of claim 1, wherein the lexicon generator further comprises a confidence assignor, for each word sequence identified in the messages of a selected context, a confidence score based upon the count of occurrences of the word sequence among the messages of the context;

and

wherein the lexicon generator further generates the lexicon based upon the confidence scores assigned by the confidence assignor component to respective word sequences of the context.

6. The computer system of claim 5, wherein the confidence assignor assigns the confidence scores for the respective messages by:

for respective word sequences, identifying the count of the occurrences of the word sequence among the messages of the respective contexts over time; and

assigning the confidence scores to the respective word sequences according to the count over time of the occurrences of the respective word sequences among the messages of the respective contexts.

7. The computer system of claim 6, wherein the lexicon generator generates the lexicon for the context by choosing the word sequences having confidence scores above a confidence score threshold.

8. The computer system of claim 6, wherein identifying the count of the occurrences of the word sequence over time further comprises:

identifying, over time, a spike in the count of the occurrences of the word sequence that indicates a trend in the occurrences of the word sequence over time.

9. The computer system of claim 6, wherein the lexicon generator generates the lexicon for the context by:

ordering the respective word sequences within the context according to the respective confidence scores; and

choosing a top percentage of the word sequences having highest confidence scores among the messages of the context.

10. The computer system of claim 1, wherein the lexicon generator further removes uniquely identifying information from the respective messages that identifies the generators or sharers of the messages.

11. The computer system of claim 10, wherein:

the lexicon generator uses the uniquely identifying information to identify the context of the message; and

the lexicon generator excludes word sequences that include the uniquely identifying information.

12. The computer system of claim 11, wherein:

the social network interface retrieves social network user profiles of the respective generators or sharers of the respective messages; and

the lexicon generator retrieves the uniquely identifying information from the social network user profiles of the respective generators or sharers of the messages.

13. The computer system of claim 1, wherein the lexicon presenter the lexicon of word sequence to the user by:

receiving, from the user, a context query that specifies context properties;

among the contexts, identifying a selected context that matches the context properties of the query; and

presents the lexicon of word sequences of messages for the selected context.

14. The method of claim 1, wherein identifying the selected word sequence in the lexicon by which at least one user of the social network who is associated with the selected word sequence is identifiable further comprises:

determining that the selected word sequence uniquely identifies at least one user.

15. The method of claim 1, wherein identifying the selected word sequence in the lexicon by which at least one user of the social network who is associated with the selected word sequence is identifiable further comprises:

determining that the selected word sequence is associated with private information of at least one user.

16. The method of claim 1, wherein identifying the selected word sequence in the lexicon by which at least one user of the social network who is associated with the selected word sequence is identifiable further comprises:

determining that the selected word sequence identifies a social relationship between a first user of the social network and a second user of the social network.

17. A method of generating a lexicon for a context of a social network, the method involving a device and comprising:

executing, by the processor, instructions that cause the device to;

among messages of the social network, identify selected messages that are within the context;

scan the respective selected message to identify a set of word sequences;

count an occurrence of the respective word sequences among the selected messages within the context;

generate a lexicon for the context that comprises the word sequences identified in the selected messages with a higher count than a word sequence count threshold;

identify a selected word sequence by which at least one user of the social network who is associated with the selected word sequence is identifiable; and

refrain from including the selected word sequence in the lexicon; and

responsive to a request by the user to present word sequences that frequently occur in the context, present to the user the lexicon of word sequences for the context.

18. The method of claim 17, wherein:

scanning the respective selected message to identify a set of word sequences further comprises;

determining a recency of the message; and

generating the lexicon further comprises;

generate a lexicon for the context that comprises the word sequences identified in the selected messages with a higher count than a word sequence count threshold and that have a recency greater than a recency threshold.

19. The method of claim 18, wherein generating the lexicon further comprises:

generate a lexicon for the context that comprises new word sequences that are identified in the selected messages with a higher count than a word sequence count threshold, and that do not appear in messages having a recency before a recency threshold.

20. A method of generating a lexicon from users of a social network who are respectively identified by a user profile, the method comprising:

for respective users of the social network, performing an evaluation of the user profile of the user to identify a context of the user and respective messages associated with the user, while refraining from using information that distinctively identifies the selected user;

among the messages of the social network, identifying selected messages that are within the context according to the context of at least one user who is associated with the message;

scanning the respective selected message to identify a set of word sequences;

counting an occurrence of the respective word sequences among the selected messages within the context;

generating a lexicon for the context that comprises the word sequences identified in the selected messages with a higher count than a word sequence count threshold; and

responsive to a request by the user to present word sequences that frequently occur in the context, presenting to the user the lexicon of word sequences for the context.