QUERY RESPONSE USING MEDIA CONSUMPTION HISTORY

• US 20150052128A1
• Filed: 03/18/2014
• Published: 02/19/2015
• Est. Priority Date: 08/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew
• Granted Patent: US 9,477,709 B2
• Time in Patent Office: Days
• US Class Current: 707/727
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G10L 25/54 for retrieval

Abstract

Methods, systems, and apparatus for receiving a natural language query of a user, and environmental data, identifying a media item based on the environmental data, determining an entity type based on the natural language query, selecting an entity associated with the media item that matches the entity type, selecting, from a media consumption database that identifies media items that have been indicated as consumed by the user, one or more media items that have been indicated as consumed by the user and that are associated with the selected entity, and providing a response to the query based on selecting the one or more media items that have been indicated as consumed by the user and that are associated with the selected entity.

Claims

1. A computer-implemented method comprising:

receiving (i) a natural language query that references a request relating to a media item, and (ii) environmental data;

identifying a particular media item based on detecting a match between one or more features of the environmental data and one or more features of the particular media item;

determining that the particular media item is referenced in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that is (i) associated with the particular media item, and (ii) requested by the natural language query; and

providing a response to the natural language query that includes at least a portion of the information accessed at the media consumption database.

2. The method of claim 1, further comprising:

selecting a media item type based on the natural language query that references the request relating to the media item; and

identifying the particular media item based on (i) detecting the match between one or more features of the environmental data and one or more features of the particular media item, and (ii) the particular media item being associated with a media item type that corresponds to the identified media item type.

3. The method of claim 2, wherein identifying the media item type based on the natural language query that references the request relating to the media item comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with media item types; and

identifying the media item type based on determining that the transcription of the natural language query matches a particular keyword phrase that is associated with the identified media item type.

4. The method of claim 1, wherein identifying the particular media item based on detecting the match between one or more features of the environmental data and one or more features of the particular media item comprises:

obtaining audio fingerprints of the environmental data;

comparing the audio fingerprints of the environmental data to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the environmental data match audio fingerprints of the particular media item.

5. The method of claim 1, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with information types; and

accessing the information that is (i) associated with the particular media item, and (ii) requested by the natural language query, based on determining that the transcription of the natural language query matches a particular keyword phrase associated with a particular information type.

6. The method of claim 1, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises accessing information that indicates at least one of a location, a time, a date, a method of consumption of the particular media item, or entities associated with the particular media item.

7. The method of claim 1, wherein the environmental data is one of audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving (i) a natural language query that references a request relating to a media item, and (ii) environmental data;

identifying a particular media item based on detecting a match between one or more features of the environmental data and one or more features of the particular media item;

determining that the particular media item is referenced in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that is (i) associated with the particular media item, and (ii) requested by the natural language query; and

providing a response to the natural language query that includes at least a portion of the information accessed at the media consumption database.

9. The system of claim 8, wherein the operations further comprise:

selecting a media item type based on the natural language query that references the request relating to the media item; and

identifying the particular media item based on (i) detecting the match between one or more features of the environmental data and one or more features of the particular media item, and (ii) the particular media item being associated with a media item type that corresponds to the identified media item type.

10. The system of claim 9, wherein identifying the media item type based on the natural language query that references the request relating to the media item comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with media item types; and

identifying the media item type based on determining that the transcription of the natural language query matches a particular keyword phrase that is associated with the identified media item type.

11. The system of claim 8, wherein identifying the particular media item based on detecting the match between one or more features of the environmental data and one or more features of the particular media item comprises:

obtaining audio fingerprints of the environmental data;

comparing the audio fingerprints of the environmental data to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the environmental data match audio fingerprints of the particular media item.

12. The system of claim 8, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with information types; and

accessing the information that is (i) associated with the particular media item, and (ii) requested by the natural language query, based on determining that the transcription of the natural language query matches a particular keyword phrase associated with a particular information type.

13. The system of claim 8, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises accessing information that indicates at least one of a location, a time, a date, a method of consumption of the particular media item, or entities associated with the particular media item.

14. The system of claim 8, wherein the environmental data is one of audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.

15. A computer-readable storage device encoded with a computer program, the program comprising instructions that, if executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving (i) a natural language query that references a request relating to a media item, and (ii) environmental data;

identifying a particular media item based on detecting a match between one or more features of the environmental data and one or more features of the particular media item;

determining that the particular media item is referenced in a media consumption database that identifies media items that are identified as having been previously consumed by the user;

accessing, at the media consumption database, information that is (i) associated with the particular media item, and (ii) requested by the natural language query; and

providing a response to the natural language query that includes at least a portion of the information accessed at the media consumption database.

16. The computer-readable device of claim 15, the operations further comprising:

selecting a media item type based on the natural language query that references the request relating to the media item; and

identifying the particular media item based on (i) detecting the match between one or more features of the environmental data and one or more features of the particular media item, and (ii) the particular media item being associated with a media item type that corresponds to the identified media item type.

17. The computer-readable device of claim 16, wherein identifying the media item type based on the natural language query that references the request relating to the media item comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with media item types; and

identifying the media item type based on determining that the transcription of the natural language query matches a particular keyword phrase that is associated with the identified media item type.

18. The computer-readable device of claim 15, wherein identifying the particular media item based on detecting the match between one or more features of the environmental data and one or more features of the particular media item comprises:

obtaining audio fingerprints of the environmental data;

comparing the audio fingerprints of the environmental data to audio fingerprints of the particular media item; and

determining that the audio fingerprints of the environmental data match audio fingerprints of the particular media item.

19. The computer-readable device of claim 15, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises:

obtaining a transcription of the natural language query;

comparing the transcription of the natural language query to one or more keyword phrases that are associated with information types; and

accessing the information that is (i) associated with the particular media item, and (ii) requested by the natural language query, based on determining that the transcription of the natural language query matches a particular keyword phrase associated with a particular information type.

20. The computer-readable device of claim 15, wherein accessing information that is (i) associated with the particular media item, and (ii) requested by the natural language query comprises accessing information that indicates at least one of a location, a time, a date, a method of consumption of the particular media item, or entities associated with the particular media item.