Query response using media consumption history

• US 9,477,709 B2
• Filed: 03/18/2014
• Issued: 10/25/2016
• Est. Priority Date: 08/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew
• Primary Examiner(s): HICKS, MICHAEL J
• Application Number:
  US14/217,940
• Publication Number:
  US 20150052128A1
• Time in Patent Office: 952 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G10L 25/54 for retrieval

Abstract

Methods, systems, and apparatus for receiving a natural language query of a user, and environmental data, identifying a media item based on the environmental data, determining an entity type based on the natural language query, selecting an entity associated with the media item that matches the entity type, selecting, from a media consumption database that identifies media items that have been indicated as consumed by the user, one or more media items that have been indicated as consumed by the user and that are associated with the selected entity, and providing a response to the query based on selecting the one or more media items that have been indicated as consumed by the user and that are associated with the selected entity.

Claims

1. A computer-implemented method comprising:

receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;

identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;

determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and

providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.

2. The method of claim 1, further comprising:

selecting a media item type based on the natural language query submitted by the user; and

identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.

3. The method of claim 2, wherein selecting the media item type based on the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and

selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.

4. The method of claim 1, wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:

obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;

comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.

5. The method of claim 1, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and

accessing, at the media consumption database, the information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.

6. The method of claim 1, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.

7. The method of claim 1, wherein the ambient environmental data obtained from the environment of the user is one of ambient audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;

identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;

determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and

providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.

9. The system of claim 8, wherein the operations further comprise:

selecting a media item type based on the natural language query submitted by the user; and

identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.

10. The system of claim 9, wherein selecting the media item type based on the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and

selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.

11. The system of claim 8, wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:

obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;

comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.

12. The system of claim 8, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and

accessing, at the media consumption database, information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.

13. The system of claim 8, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.

14. The system of claim 8, wherein the ambient environmental data obtained from the environment of the user is one of ambient audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.

15. A computer-readable storage device encoded with a computer program, the program comprising instructions that, if executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;

identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;

determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and

providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.

16. The computer-readable device of claim 15, the operations further comprising:

selecting a media item type based on the natural language query submitted by the user; and

identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.

17. The computer-readable device of claim 16, wherein selecting the media item type based on the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and

selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.

18. The computer-readable device of claim 15, wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:

obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;

comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.

19. The computer-readable device of claim 15, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:

obtaining a transcription of the natural language query submitted by the user;

determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and

accessing, at the media consumption database, the information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.

20. The computer-readable device of claim 15, wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.