Providing recommendations using information determined for domains of interest

• US 8,429,106 B2
• Filed: 12/11/2009
• Issued: 04/23/2013
• Est. Priority Date: 12/12/2008
• Status: Expired due to Fees

• Current Assignee: Veritone Alpha, Inc. (Veritone Media)
• Original Assignee: Atigeo LLC
• Inventors: Downs, Oliver B., Sandoval, Michael, Branzan, Claudiu Alin, Iovanov, Vlad Mircea, Khalsa, Sopurkh Singh
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US12/636,630
• Publication Number:
  US 20100153324A1
• Time in Patent Office: 1,229 Days
• Field of Search: 706/55
• US Class Current: 706/21
• CPC Class Codes: G06F 40/258 Heading extraction; Automatic titling; Numbering
• CPC Class Codes: G06F 40/42 Data-driven translation
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Techniques are described for determining and using information related to domains of interest, such as by automatically analyzing documents and other information related to a domain in order to automatically determine relationships between particular terms within the domain. Such automatically determined information may then be used to assist users in obtaining information from the domain that is of interest (e.g., documents with contents that are relevant to user-specified terms and/or to other terms that are determined to be sufficiently related to the user-specified terms). For example, recommendations may be automatically generated for a user by using information about specified preferences or other interests of the user with respect to one or more terms and identifying other particular terms that are sufficiently probable to be of interest to that user, such as based on a generated probabilistic representation of relationships between particular terms for the domain.

Claims

1. A computer-implemented method for providing information based on automatically determined relationships, the method comprising:

under control of one or more computing systems configured to provide a relevance determination service, automatically determining relevant information to recommend by,

automatically analyzing contents of a plurality of documents related to a first domain of interest to identify multiple inter-term relationships between at least some of a plurality of terms that are present in the contents of the documents, each of the identified relationships indicating an initial assessed relevance between at least one of the terms and at least one other of the terms;

automatically generating a term relevance neural network that models the assessed relevances of the identified relationships, the term relevance neural network initially modeling the assessed initial relevances, and repeatedly updating the assessed relevances that are modeled by the term relevance neural network based on feedback obtained from users that perform selections corresponding to the plurality of terms;

automatically generating a probabilistic Bayesian network based on the updated assessed relevances of at least some of the identified relationships, the probabilistic Bayesian network including information that indicates probabilities for relationships between at least some of the plurality of terms; and

using the information included in the probabilistic Bayesian network to provide recommendations related to the first domain by, for each of multiple users;

obtaining information about a first group of one or more of the plurality of terms for which the user has expressed a preference;

for each of one or more target terms of the plurality of terms that are not in the first group, automatically determining a probability that the target term is an unexpressed preference of the user, the determined probability being based on the preference of the user for the one or more terms of the first group and being based on one or more relationships between the one or more terms of the first group and the target term that are indicated in the information included in the probabilistic Bayesian network; and

providing one or more recommendations for the user related to the first domain that are based on a selected second group of at least one of the target terms, the target terms of the second group being selected based on the determined probabilities that those target terms are unexpressed preferences of the user, and wherein the target terms of the selected second group for at least one of the multiple users differ from the target terms of the selected second group for at least one other of the multiple users.

2. The method of claim 1 wherein the automatic generating of the probabilistic Bayesian network includes, for each of the at least some identified relationships, determining a direction of influence between the at least one term and the at least one other term of the identified relationship, and determining one or more conditional probabilities that represent a strength of the influence between the at least one term and the at least one other term of the identified relationship, and wherein the information included in the probabilistic Bayesian network includes the determined directions of influence and the determined conditional probabilities.

3. The method of claim 2 wherein the automatic generating of the probabilistic Bayesian network further includes generating multiple decision trees that each correspond to a portion of the information included in the probabilistic Bayesian network, and wherein the automatic determining of the probability for each of the target terms for one of the multiple users is performed using the generated multiple decision trees.

4. The method of claim 1 wherein the using of the information included in the probabilistic Bayesian network to provide recommendations related to the first domain to the multiple users further includes, after the information included in the probabilistic Bayesian network is used to provide recommendations related to the first domain to one or more of the multiple users:

obtaining information about actual preferences of additional users related to the plurality of terms, the additional users being distinct from the multiple users; and

updating the information included in the probabilistic Bayesian network to reflect the obtained information about the actual preferences of the additional users;

and wherein the using of the information included in the probabilistic Bayesian network to provide recommendations related to the first domain to the multiple users other than the one or more users includes using the updated information.

5. The method of claim 1 further comprising:

automatically generating a second probabilistic Bayesian network that indicates second probabilities for relationships between the plurality of terms for the first domain and that is based on obtained information about actual preferences of a plurality of users for the plurality of terms; and

for each of one or more of the multiple users, after the obtaining of the information about the first group of the one or more terms for which the user has expressed a preference, using the second probabilistic Bayesian network to automatically determining a second probability for each of the one or more target terms that the target term is an unexpressed preference of the user, the determined second probability for at least one of the target terms being distinct from the probability for that target term for the user that was determined based on the probabilistic Bayesian network generated using the term relevance neural network; and

wherein the selected second group of target terms that is used to provide recommendations to at least one of the one or more users further includes at least one target term that is selected based on the determined second probabilities from the second probabilistic Bayesian network and that is not selected based on the determined probabilities from the probabilistic Bayesian network generated using the term relevance neural network.

6. The method of claim 1 wherein the automatic determining of the relevant information to recommend further includes:

automatically analyzing contents of other documents related to a second domain of interest to identify multiple additional relationships between a second plurality of terms that are present in the contents of the other documents, the second plurality of terms including one or more first terms that are part of the plurality of terms present in the contents of the documents related to the first domain and including one or more other second terms that are not part of the plurality of terms present in the contents of the documents related to the first domain, and the identified multiple additional relationships indicating an initial assessed relevance of one or more of the first terms to one or more of the second terms;

automatically updating the probabilistic Bayesian network to include additional information that indicates probabilities corresponding to at least one of the additional relationships, the automatic updating being performed based at least in part on the initial assessed relevance of the one or more first terms to the one or more second terms but without any information of a preference of any users for the one or more second terms; and

after obtaining information about a first group of one or more terms for which a first user has a preference but that do not include any of the second terms, providing one or more recommendations to the first user that are based on at least one of the second terms, the at least one second terms being automatically selected based on a determined probability that the at least one second terms are an unexpressed preference of the first user, the determined probability that the at least one second terms are an unexpressed preference of the first user being determined based on the additional information included in the updated probabilistic Bayesian network.

7. The method of claim 1 wherein, for one of the multiple users, the relevance determination service has a cold start recommendation problem based on a lack of information to enable identification of any other users having similarity to the one user to enable the relevance determination service to provide recommendations to the one user related to the first domain based on available information about the other users, and wherein the identified relationships from the automatic analyzing of the contents of the documents related to the first domain are used by the relevance determination service to overcome the cold start recommendation problem with respect to the providing of the one or more recommendations to the one user.

8. The method of claim 7 wherein the one or more recommendations provided to the one user include at least one of one or more of the plurality of documents related to the first domain and of one or more of the target terms selected for the second group of the one user.

9. The method of claim 1 wherein the relevance determination service is accessible to users via one or more affiliated services, such that at least some providing of the recommendations for the multiple users includes providing those recommendations to the affiliated services which further provide information based on those recommendations to those users, and wherein the relevance determination service is a fee-based service that obtains fees from the affiliated services and/or from those users.

10. A computer-implemented method for providing information based on automatically determined relationships, the method comprising:

under control of one or more computing systems configured to provide recommendation information based on automatically determined relationships between terms;

receiving one or more indications of a group of multiple content items whose contents are representative of a subject area of interest, the contents including a plurality of terms;

automatically analyzing, by the configured one or more computing systems, the multiple content items of the group to identify relationships between at least some of the plurality of terms, wherein the relationships identified by the automatic analyzing include multiple inter-term relationships that are each between at least two of the plurality of terms, wherein the automatic analyzing further includes automatically assessing for each of the multiple inter-term relationships an initial degree of relevance between the at least two terms for the inter-term relationship, wherein a first of the identified multiple inter-term relationships has an assessed degree of relevance of one or more first terms of the plurality of terms to one or more other second terms of the plurality of terms, and wherein the automatic analyzing of the multiple content items of the group further includes generating probabilistic information for the first inter-term relationship that indicates a likelihood of a relationship between the one or more first terms and the one or more other second terms;

obtaining information about one or more indicated terms for which a first user has a preference, the indicated terms including at least one of the first terms but not including any of the second terms;

for each of one or more of the second terms, automatically determining, by the configured one or more computing systems, a likelihood that the second term is of interest to the first user based at least in part on the at least one term included in the indicated terms and on the assessed degree of relevance of the one or more first terms to the one or more second terms, wherein the automatic determining of the likelihood for each of the one or more second terms is based at least in part on use of the generated probabilistic information; and

providing an indication of at least one of the one or more second terms that is selected to enable one or more recommendations to be provided to the first user based on the at least one second terms, the at least one second terms being selected based on one or more determined criteria for assessing the determined likelihoods of the at least one second terms.

11. The method of claim 10 wherein the generated probabilistic information includes a Bayesian network having included information that indicates for each of at least some of the multiple inter-term relationships a direction of influence between the at least two terms for the inter-term relationship and that indicates conditional probability information for each of the at least some inter-term relationships.

12. The method of claim 11 further comprising, after the providing of the indication of the at least one selected second term:

obtaining feedback from users related to the plurality of terms;

automatically updating the conditional probability information indicated in the information included in the Bayesian network for one or more inter-term relationships based at least in part on the obtained additional feedback; and

using the updated conditional probability information included in the Bayesian network to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

13. The method of claim 10 wherein the generated probabilistic information includes one or more decision trees that each represents one or more of at least some of the multiple inter-term relationships and includes at least one decision node and multiple end nodes, each of the end nodes representing a probability of a user having an interest in one of the at least terms for one of the one or more represented inter-term relationships.

14. The method of claim 13 further comprising, after the providing of the indication of the at least one selected second term:

obtaining feedback from users related to the plurality of terms;

automatically updating the probability represented by one or more end nodes of one or more of the decision trees based at least in part on the obtained additional feedback; and

using the updated represented probability of the one or more decision trees to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

15. The method of claim 10 wherein the automatic analyzing further includes generating a term relevance neural network that represents the initial assessed degrees of relevance between the at least two terms for the multiple inter-term relationships, and repeatedly updating the assessed degrees of relevance for the multiple inter-term relationships that are represented by the term relevance neural network based on feedback obtained from users that perform selections corresponding to the plurality of terms, and wherein the generating of the probabilistic information is based on the updated assessed degree of relevance for the first identified relationship.

16. The method of claim 15 further comprising, after the providing of the indication of the at least one selected second term:

obtaining additional feedback from users related to the plurality of terms;

automatically identifying one or more additional inter-term relationships based at least in part on the obtained additional feedback;

automatically generating a new term relevance neural network that represents an initial assessed degree of relevance for the identified one or more additional inter-term relationships and that represents the updated assessed degrees of relevance for one or more of the multiple inter-term relationships;

automatically generating a new probabilistic representation of the at least some inter-term relationships and of at least one of the additional inter-term relationships; and

using information included in the generated new probabilistic representation to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

17. The method of claim 10 further comprising, after the generating of the probabilistic information:

obtaining feedback from users related to the plurality of terms;

automatically updating the information included in the generated probabilistic information for one or more inter-term relationships by combining the obtained additional feedback with the assessed degrees of relevance from the automatic analyzing of the multiple content items of the group, the combining including using differing weights for the obtained additional feedback and the assessed degrees of relevance from the automatic analyzing of the multiple content items of the group; and

using the updated included information to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

18. The method of claim 17 further comprising, after the using of the updated included information to automatically determine the likelihoods of the one or more additional users having an interest in the one or more additional terms, obtaining additional feedback from users related to the plurality of terms, and automatically determining to use the obtained additional feedback from the users in lieu of the assessed degrees of relevance from the automatic analyzing of the multiple content items of the group as part of automatically determining further likelihoods of one or more other users having an interest in one or more terms.

19. The method of claim 10 further comprising automatically generating second probabilistic information for the first inter-term relationship based at least in part on information obtained about actual preferences of a plurality of users, the second probabilistic information indicating a likelihood of the relationship between the one or more first terms and the one or more other second terms, and wherein the automatic determining of the likelihood for at least one of the one or more second terms is further based on use of the second probabilistic information.

20. A non-transitory computer-readable storage medium having stored contents that configure a computing system of a relevance determination system to provide information based on automatically determined relationships, by performing a method comprising:

under control of the configured computing system,

automatically analyzing, by the configured computing system, contents of multiple related content items in order to identify relationships between at least some of a plurality of terms included in the contents, wherein a first of the identified relationships indicates an assessed relevance of a first term of the plurality of terms to one or more other second terms of the plurality of terms, wherein the contents of the multiple related content items are representative of a subject area of interest;

obtaining information about one or more indicated terms of interest to a first user, wherein the indicated terms include the first term but not including any of the second terms, and wherein the one or more terms are indicated by the first user to be preferences of the first user;

automatically determining, by the configured computing system, a likelihood that a selected one of the second terms is of interest to the first user based at least in part on the assessed relevance of the first term to the one or more second terms, wherein the automatic determining of the likelihood is performed for each of the one or more other second terms; and

providing an indication of the selected one second term and of the determined likelihood to enable one or more suggestions to be determined for the first user based on the one second term, wherein the selected one second term is selected based on one or more criteria for assessing the determined likelihoods of the one or more other second terms, and wherein the providing of the indication of the selected one second term and of the determined likelihood includes determining the one or more suggestions based on the selected one second term and providing at least one of the determined suggestions as a recommendation to the first user.

21. The non-transitory computer-readable storage medium of claim 20 wherein the recommendation provided to the first user is the one second term.

22. The non-transitory computer-readable storage medium of claim 20 wherein the computer-readable storage medium is a memory of the configured computing system on which the contents are stored, and wherein the stored contents are instructions that, when executed, program the configured computing system to perform the method.

23. The non-transitory computer-readable storage medium of claim 20 wherein the automatic determining of the likelihood is based on generated probabilistic information for the first identified relationship that indicates one or more likelihoods of one or more relationships between the first term and the one or more other second terms.

24. The non-transitory computer-readable storage medium of claim 23 wherein the generated probabilistic information includes a Bayesian network having included information that indicates a direction of influence between the first term and the one or more other second terms and that indicates conditional probability information for the first identified relationship, and wherein the automatic analyzing of the contents of the multiple related content items includes generating the generated probabilistic information.

25. The non-transitory computer-readable storage medium of claim 24 wherein the method further comprises, after the providing of the indication of the selected one second term:

obtaining feedback from users related to the plurality of terms;

automatically updating the conditional probability information based at least in part on the obtained additional feedback; and

using the updated conditional probability information to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

26. The non-transitory computer-readable storage medium of claim 23 wherein the generated probabilistic information includes a decision tree that represents the first identified relationship and that has at least one decision node and multiple end nodes, each of the end nodes representing a probability of a user having an interest in one of the one or more other second terms, and wherein the automatic analyzing of the contents of the multiple related content items includes generating the generated probabilistic information.

27. The non-transitory computer-readable storage medium of claim 26 wherein the method further comprises, after the providing of the indication of the selected one second term:

obtaining feedback from users related to the plurality of terms;

automatically updating the probability represented by one or more of the end nodes based at least in part on the obtained additional feedback; and

using the updated represented probability to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

28. The non-transitory computer-readable storage medium of claim 23 wherein the automatic analyzing further includes generating a term relevance neural network that represents one or more initial assessed degrees of relevance between the first term and the one or more other second terms, and repeatedly updating the assessed degrees of relevance based on feedback obtained from users performing selections corresponding to the plurality of terms, and wherein the generated probabilistic information is based on the updated assessed degrees of relevance.

29. The non-transitory computer-readable storage medium of claim 28 wherein the method further comprises, after the providing of the selected one second term:

obtaining additional feedback from users related to the plurality of terms;

automatically identifying one or more additional inter-term relationships based at least in part on the obtained additional feedback;

automatically generating a new term relevance neural network that represents an assessed degree of relevance for the identified one or more additional inter-term relationships and that represents the updated assessed degrees of relevance;

automatically generating a new probabilistic representation of at least one of the additional inter-term relationships; and

using information included in the generated new probabilistic representation to automatically determine likelihoods of one or more additional users having an interest in one or more additional terms of the plurality of terms.

30. The non-transitory computer-readable storage medium of claim 23 wherein the method further comprises automatically generating second probabilistic information for the first identified relationship based at least in part on information obtained about actual preferences of a plurality of users, the second probabilistic information indicating one or more additional likelihoods of the one or more relationships between the first term and the one or more other second terms, and wherein the automatic determining of the likelihood for the selected one second term is further based on use of the second probabilistic information.

31. The non-transitory computer-readable storage medium of claim 20 wherein the providing of the indication of the second one second term further includes providing an indication of the determined likelihood.

32. The non-transitory computer-readable storage medium of claim 20 wherein the one or more indicated terms of interest are search terms specified by the first user, and wherein the providing of the indication of the selected one second term includes generating search results that are based at least in part on the selected one second term and are used in the providing of the at least one determined suggestions.

33. The non-transitory computer-readable storage medium of claim 32 wherein the generated search results include one or more of the multiple related content items.

34. The non-transitory computer-readable storage medium of claim 32 wherein the generated search results include one or more content items that are related to the subject area of interest but are not part of the multiple related content items.

35. The non-transitory computer-readable storage medium of claim 20 wherein the one or more indicated terms of interest are specified by the first user, and wherein the provided at least one determined suggestion includes the selected one second term for display to the first user to enable the first user to specify the selected one second term as being a further preference of the first user.

36. The non-transitory computer-readable storage medium of claim 20 wherein the one or more indicated terms of interest are specified by the first user, and wherein the provided at least one determined suggestion includes the selected one second term for display to the first user to enable the first user to specify the selected one second term as being part of a common multi-term theme with the one or more indicated terms of interest for the subject area of interest.

37. The non-transitory computer-readable storage medium of claim 20 wherein the automatic determining of the likelihood for the selected one second term is performed without using any information about any actual preferences of any users related to the one or more other second terms.

38. The non-transitory computer-readable storage medium of claim 20 wherein the method further comprises:

automatically analyzing a plurality of additional content items to identify one or more additional inter-term relationships related to a second subject area of interest to which the plurality of additional content items relate, the second subject area of interest being distinct from the subject area of interest and the plurality of additional content items being distinct from the multiple related content items, the identified additional relationships including a second relationship between the first term and one or more other third terms that are not part of the plurality of terms; and

providing an indication of at least one of the third terms that is selected to enable one or more additional suggestions to be provided to the first user based on the at least one third term, the at least one third terms being selected without using any information about any actual preferences of any users related to the one or more third terms.

39. The non-transitory computer-readable storage medium of claim 20 further comprising automatically analyzing a plurality of additional content items related to a second subject area of interest to identify one or more of the plurality of additional content items that are related to the selected one second term, and wherein the provided at least one suggestion includes the identified one or more additional content items related to the selected one second term.

40. The non-transitory computer-readable storage medium of claim 20 wherein the selected one second term is selected to represent an unexpressed preference of the first user.

41. The non-transitory computer-readable storage medium of claim 20 wherein the multiple related content items include at least one of textual documents whose text contains at least some of the plurality of terms, audio information, image information, video information, biological information, alphanumeric data structures, symbolic data structures, and mathematical data structures, and wherein the configured computing system is part of a relevance determination system that performs the providing of the at least one determined suggestion based on automatically determined relationships between terms.

42. A computing system configured to provide information based on automatically determined relationships, comprising:

one or more processors; and

a relevance determination system that is configured to, when executed by at least one of the one or more processors, provide information based on automatically determined relationships by;

automatically analyzing multiple content items that are representative of a subject area of interest in order to identify inter-term relationships between a plurality of terms included in contents of the multiple content items, and wherein each of the inter-term relationships indicates an assessed relevance of at least one first term of the plurality of terms to at least one other second term of the plurality of terms;

automatically generating a probabilistic representation of selected inter-term relationships based at least in part on the assessed relevances for the selected inter-term relationships, the probabilistic representation including information related to a determined likelihood of a relationship between the at least one first term and the at least one second term for each of the selected inter-term relationships; and

providing information about the determined likelihood of the relationship between the at least one first term and the at least one second term for at least one of the selected inter-term relationships to enable one or more suggestions to be determined for a user who has an interest in the at least one first term for the at least one selected inter-term relationship, wherein the providing of the information about the determined likelihood of the relationship between the at least one first term and the at least one second term for the at least one selected inter-term relationship includes;

after obtaining information about the user having a preference for the at least one first term of the at least one selected inter-term relationship, using the information included in the generated probabilistic representation to automatically determine that the at least one second term of the at least one selected inter-term relationship is also of interest to the user based at least in part on the determined likelihood for the at least one selected inter-term relationship;

determining the one or more suggestions for the user based at least in part on the at least one second term of the at least one selected inter-term relationship; and

providing the determined one or more suggestions for the user.

43. The computing system of claim 42 wherein the relevance determination system includes software instructions for execution by the at least one processors.

44. The computing system of claim 42 wherein the relevance determination system consists of a means for performing the providing of the information based on the automatically determined relationships.

45. A system configured to provide information based on automatically determined relationships, comprising:

one or more processors of one or more computing systems;

a relevance determination system that is configured to, when executed by at least one of the one or more processors, provide information based on automatically determined relationships by;

automatically analyzing multiple content items related to a subject area of interest in order to identify inter-term relationships between a plurality of terms related to the multiple content items, each of the inter-term relationships indicating an assessed relevance of at least one first term of the plurality of terms to at least one other second term of the plurality of terms;

automatically generating a probabilistic representation of selected inter-term relationships based at least in part on the assessed relevances for the selected inter-term relationships, the probabilistic representation including information related to a determined likelihood of a relationship between the at least one first term and the at least one second term for each of the selected inter-term relationships; and

providing information about the determined likelihood of the relationship between the at least one first term and the at least one second term for at least one of the selected inter-term relationships to enable one or more suggestions to be determined for a user who has an interest in the at least one first term for the at least one selected inter-term relationship; and

one or more additional systems configured to, when executed by at least one of the one or more processors, receive the provided information about the determined likelihood of the relationship between the at least one first term and the at least one second term for at least one of the selected inter-term relationships, and to, for each of multiple users;

obtain information about one or more terms indicated by the user to be preferences of the user;

automatically determine one or more second terms that are of likely interest to the user based at least in part on the received provided information;

automatically determine one or more recommendations for the user based at least in part on the determined one or more second terms; and

provide the determined one or more recommendations to the user.

46. The system of claim 45 wherein, for one of the multiple users, the automatic determining of the one or more second terms that are of likely interest to the one user includes determining for each of multiple second terms a probability that the one user has an interest in the second term, and selecting the determined one or more second terms from the multiple second terms based on one or more determined criteria for assessing the determined probabilities of the multiple second terms.