INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA SEARCH AND PLAYBACK

• US 20170068670A1
• Filed: 12/08/2015
• Published: 03/09/2017
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: SIDKI, Raumi N., ORR, Ryan M., LISTER, Patrick M., RUSSELL, Jonathan H.
• Granted Patent: US 10,740,384 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/438 Presentation of query results
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739

Abstract

Systems and processes are disclosed for operating a digital assistant in a media environment. In an example process, a primary set of media items can be displayed. An audio input containing a media-related request can be received. A primary user intent corresponding to the media-related request can be determined. In accordance with a determination that the primary user intent comprises a user intent to narrow the primary media search query, a second primary media search query corresponding to the primary user intent can be generated. The second primary media search query can be based on the media-related request and the primary media search query. The second primary media search query can be performed to obtain a second primary set of media items. Display of the primary set of media items can be replaced with display of the second primary set of media items.

Claims

1. A non-transitory computer-readable medium storing instructions for operating a digital assistant of a media system, the instructions, when executed by one or more processors, cause the one or more processors to:

display a primary set of media items on a display;

in response to detecting a user input, receive audio input containing a media-related request in natural language speech form;

determine a primary user intent corresponding to the media-related request;

determine whether the primary user intent comprises a user intent to narrow a primary media search query corresponding to the primary set of media items; and

in accordance with a determination that the primary user intent comprises a user intent to narrow the primary media search query;

generate, based on the media-related request and the primary media search query, a second primary media search query that corresponds to the primary user intent;

perform the second primary media search query to obtain a second primary set of media items; and

replace display of the primary set of media items with display of the second primary set of media items.

2. The non-transitory computer-readable medium of claim 1, wherein determining whether the primary user intent comprises a user intent to narrow the primary media search query comprises:

determining whether the media-related request includes a word or phrase corresponding to a user intent to narrow the primary media search query.

3. The non-transitory computer-readable medium of claim 1, wherein the second primary media search query includes one or more parameter values defined in the media-related request and one or more parameter values of the primary media search query.

4. The non-transitory computer-readable medium of claim 1, wherein the second primary media search query includes a set of parameter values, and wherein the instructions further cause the one or more processors to:

identify a core set of parameter values from the set of parameter values, the core set of parameter values having fewer parameter values than the set of parameter values;

generate one or more additional media search queries based on the core set of parameter values;

perform the one or more additional media search queries to obtain one or more additional sets of media items; and

display the one or more additional sets of media items on the display.

5. The non-transitory computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the primary user intent does not comprise a user intent to narrow the primary media search query;

determine whether the primary user intent comprises a user intent to perform a new media search query; and

in accordance with a determination that the primary user intent comprises a user intent to perform a new media search query;

generate, based on the media-related request, a third primary media search query that corresponds to the primary user intent;

determine whether at least one media item corresponding to the third primary media search query can be obtained; and

in accordance with a determination that at least one media item corresponding to the third primary media search query can be obtained;

perform the third primary media search query to obtain a third primary set of media items; and

replace display of the primary set of media items with display of the third primary set of media items.

6. The non-transitory computer-readable medium of claim 5, wherein determining whether the primary user intent comprises a user intent to perform a new media search query further comprises:

determining whether the media-related request includes a word or phrase corresponding to a parameter value of one or more media items.

7. The non-transitory computer-readable medium of claim 5, wherein performing the third primary media search query comprises identifying a candidate media item associated with a parameter value that is included in one or more media critic reviews of the identified candidate media item.

8. The non-transitory computer-readable medium of claim 5, wherein the instructions further cause the one or more processors to:

in accordance with a determination that no media items correspond to the third primary media search query;

identify a least pertinent parameter value of the third primary media search query;

determine, based on the identified least pertinent parameter value, one or more alternative parameter values;

perform, using the one or more alternative parameter values, one or more alternative primary media search queries to obtain a fourth primary set of media items; and

replace display of the primary set of media items with display of the fourth primary set of media items.

9. The non-transitory computer-readable medium of claim 5, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the primary user intent does not comprise a user intent to narrow the primary media search query;

determine one or more secondary user intents based on the primary user intent and one or more previous user intents, the one or more previous user intents corresponding to one or more previous media-related requests received prior to the media-related request;

generate, one or more secondary media search queries that correspond to the one or more secondary user intents;

perform the one or more secondary media search queries to obtain one or more secondary sets of media items; and

display the one or more secondary sets of media items on the display.

10. The non-transitory computer-readable medium of claim 9, wherein the instructions further cause the one or more processors to:

determine one or more combinations of the primary user intent and the one or more previous user intents, wherein each of the one or more combinations is associated with at least one media item, and wherein the one or more secondary intents comprise the one or more combinations.

11. The non-transitory computer-readable medium of claim 9, wherein the instructions further cause the one or more processors to:

receive a media search history from a second electronic device, wherein the one or more secondary user intents are generated based on the media search history received from the second electronic device.

12. The non-transitory computer-readable medium of claim 9, wherein:

a plurality of text is displayed on the display while receiving the audio input;

the plurality of text is associated with a plurality of media items displayed on the display while receiving the audio input; and

the one or more secondary user intents are generated based on the displayed plurality of text.

13. The non-transitory computer-readable medium of claim 9, wherein the instructions further cause the one or more processors to:

determine a ranking score for each of the one or more secondary user intents, wherein the one or more secondary sets of media items are displayed in accordance with the ranking score for each of the one or more secondary user intents.

14. The non-transitory computer-readable medium of claim 13, wherein the ranking score for each of the one or more secondary user intents is based on a time at which each of the media-related requests and the one or more previous media-related requests were received.

15. The non-transitory computer-readable medium of claim 5, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the primary user intent does not comprise a user intent to perform a new media search query;

determine whether the primary user intent comprises a user intent to correct a portion of the primary media search query; and

in accordance with a determination that the primary user intent comprises a user intent to correct a portion of the primary media search query;

generate, based on the media-related request and the primary media search query request, a fifth primary media search query that corresponds to the primary user intent;

perform the fifth primary media search query to obtain a fifth primary set of media items; and

replace display of the primary set of media items with display of the fifth primary set of media items.

16. The non-transitory computer-readable medium of claim 15, wherein determining whether the primary user intent comprises a user intent to correct a portion of the primary media search query comprises:

determining whether a sequence of phonemes representing a portion of the media-related request is substantially similar to a sequence of phonemes representing a portion of a previous media-related request that corresponds to the primary media search query.

17. The non-transitory computer-readable medium of claim 15, wherein generating the fifth primary media search query comprises:

identifying a set of media items associated with a portion of the primary media search query that is not to be corrected, wherein the fifth primary media search query is generated based on one or more parameter values of the set of media items associated with the portion of the primary media search query that is not to be corrected.

18. The non-transitory computer-readable medium of claim 15, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the primary user intent comprises a user intent to correct a portion of the primary media search query;

exclude the primary media search query from consideration when determining a secondary user intent corresponding to the media-related request.

19. The non-transitory computer-readable medium of claim 15, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the primary user intent does not comprise a user intent to correct a portion of the primary media search query;

determine whether the primary user intent comprises a user intent to change a focus of a user interface displayed on the display, wherein the user interface includes a plurality of media items; and

in accordance with a determination that the primary user intent comprises a user intent to change a focus of a user interface displayed on the display, change a focus of the user interface from a first media item of the plurality of media items to a second media item of the plurality of media items.

20. The non-transitory computer-readable medium of claim 19, wherein determining whether the primary user intent comprises a user intent to change a focus of a user interface displayed on the display comprises:

determining whether the media-related request includes a word or phrase corresponding to a user intent to change a focus of a user interface displayed on the display.

21. The non-transitory computer-readable medium of claim 19, wherein the user interface includes a plurality of text corresponding to the plurality of media items in the user interface, and wherein the determination of whether the primary user intent comprises a user intent to change a focus of a user interface displayed on the display is based on the plurality of text.

22. The non-transitory computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to:

while receiving the audio input;

determine a preliminary user intent based on a received portion of the audio input;

identify data that is required to fulfill the preliminary user intent;

determine whether the data is stored on the media system at a time the preliminary user intent is determined; and

in accordance with a determination that the data is not stored on the media system at the time the preliminary user intent is determined, obtaining the data.

23. A method for operating a digital assistant of a media system, the method comprising:

at one or more electronic devices comprising one or more processors and memory;

displaying a primary set of media items on a display;

in response to detecting a user input, receiving audio input containing a media-related request in natural language speech form;

determining a primary user intent corresponding to the media-related request;

determining whether the primary user intent comprises a user intent to narrow a primary media search query corresponding to the primary set of media items; and

in accordance with a determination that the primary user intent comprises a user intent to narrow the primary media search query;

generating, based on the media-related request and the primary media search query, a second primary media search query that corresponds to the primary user intent;

performing the second primary media search query to obtain a second primary set of media items; and

replacing display of the primary set of media items with display of the second primary set of media items.

24. The method of claim 23, further comprising:

in accordance with a determination that the primary user intent does not comprise a user intent to narrow the primary media search query;

determining whether the primary user intent comprises a user intent to perform a new media search query;

in accordance with a determination that the primary user intent comprises a user intent to perform a new media search query;

generating, based on the media-related request, a third primary media search query that corresponds to the primary user intent;

determining whether at least one media item corresponding to the third primary media search query can be obtained; and

in accordance with a determination that at least one media item corresponding to the third primary media search query can be obtained;

performing the third primary media search query to obtain a third primary set of media items; and

replacing display of the primary set of media items with display of the third primary set of media items.

25. The method of claim 24, further comprising:

in accordance with a determination that the primary user intent does not comprise a user intent to narrow the primary media search query;

determining one or more secondary user intents based on the primary user intent and one or more previous user intents, the one or more previous user intents corresponding to one or more previous media-related requests received prior to the media-related request;

generating, one or more secondary media search queries that correspond to the one or more secondary user intents;

performing the one or more secondary media search queries to obtain one or more secondary sets of media items; and

displaying the one or more secondary sets of media items on the display.

26. The method of claim 25, further comprising:

determining one or more combinations of the primary user intent and the one or more previous user intents, wherein each of the one or more combinations is associated with at least one media item, and wherein the one or more secondary intents comprise the one or more combinations.

27. The method of claim 24, further comprising:

in accordance with a determination that the primary user intent does not comprise a user intent to perform a new media search query;

determining whether the primary user intent comprises a user intent to correct a portion of the primary media search query;

in accordance with a determination that the primary user intent comprises a user intent to correct a portion of the primary media search query;

generating, based on the media-related request and the primary media search query request, a fifth primary media search query that corresponds to the primary user intent;

performing the fifth primary media search query to obtain a fifth primary set of media items; and

replacing display of the primary set of media items with display of the fifth primary set of media items.

28. The method of claim 27, further comprising:

in accordance with a determination that the primary user intent does not comprise a user intent to correct a portion of the primary media search query;

determining whether the primary user intent comprises a user intent to change a focus of a user interface displayed on the display, wherein the user interface includes a plurality of media items; and

in accordance with a determination that the primary user intent comprises a user intent to change a focus of a user interface displayed on the display, changing a focus of the user interface from a first media item of the plurality of media items to a second media item of the plurality of media items.

29. An electronic device for operating a digital assistant of a media system, the device comprising:

one or more processors;

memory storing instructions which, when executed by the one or more processors, cause the one or more processors to;

display a primary set of media items on a display;

in response to detecting a user input, receive audio input containing a media-related request in natural language speech form;

determine a primary user intent corresponding to the media-related request;

determine whether the primary user intent comprises a user intent to narrow a primary media search query corresponding to the primary set of media items;

in accordance with a determination that the primary user intent comprises a user intent to narrow the primary media search query;

generate, based on the media-related request and the primary media search query, a second primary media search query that corresponds to the primary user intent;

perform the second primary media search query to obtain a second primary set of media items; and

replace display of the primary set of media items with display of the second primary set of media items.