INTELLIGENT AUTOMATED ASSISTANT IN A MEDIA ENVIRONMENT

• US 20170285915A1
• Filed: 06/20/2017
• Published: 10/05/2017
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: NAPOLITANO, Lia T., HWANG, Grace H., PENHA, Henrique D., SHAW, Jeremiah D., FINO, Jorge S.
• Granted Patent: US 10,379,715 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/438 Presentation of query results
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04N 21/41265 : having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 : characterized by hardware details
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/47 : End-user applications
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739

Abstract

Systems and processes are disclosed for operating a digital assistant in a media environment. In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one approach, a plurality of exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The plurality of exemplary natural language requests can be contextually-related to the displayed content. In another approach, a user request can be received in response to detecting a user input of a second input type. A task that at least partially satisfies the user request can be performed. The performed task can depend on the nature of the user request and the content being displayed by the media device. In particular, the user request can be satisfied while reducing disruption to user consumption of media content.

Claims

1. An electronic device for operating a digital assistant of a media system comprising:

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying content on a display unit;

while displaying the content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining a determination of a user intent corresponding to the user utterance;

obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and

in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein;

the displayed content includes media content playing on the media device;

the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;

the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and

adjusting the state or the setting of the application comprises;

rewinding the media content by a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion.

2. The electronic device of claim 1, wherein the request to adjust a state or a setting of an application on the electronic device comprises a request to play a particular media item, and wherein adjusting the state or the setting of the application to satisfy the user intent comprises playing the particular media item.

3. The electronic device of claim 2, wherein the displayed content includes a user interface with a media item, wherein the user utterance does not explicitly define the particular media item to be played, and further comprising:

determining whether a focus of the user interface is on the media item; and

in accordance with a determination that a focus of the user interface is on the media item, identifying the media item as the particular media item to be played.

4. The electronic device of claim 1, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to launch the application on the electronic device.

5. The electronic device of claim 1, wherein the displayed content comprises media content playing on the electronic device, and wherein the state or the setting relates to the media content being played on the electronic device.

6. The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to fast-forward or rewind the media content playing on the electronic device.

7. The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to jump forward or backward in the media content to play a particular portion of the media content.

8. The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to pause the playing of the media content on the electronic device.

9. The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to turn-on or turn-off subtitles of the media content.

10. The electronic device of claim 1, wherein:

the displayed content includes a user interface with a first media item and a second media item;

the request to adjust a state or a setting of an application on the electronic device includes a request to switch a focus of the user interface from the first media item to the second media item; and

adjusting the state or the setting of the application to satisfy the user intent comprises switching a focus of the user interface from the first media item to the second media item.

11. The electronic device of claim 10, wherein adjusting the state or the setting of the application further comprises:

turning on closed captioning prior to restarting play of the media content from the previous portion.

12. The electronic device of claim 10, wherein:

the request to adjust a state or a setting of an application on the electronic device further comprises a request to increase a volume of the audio associated with the media content; and

adjusting the state or the setting of the application further comprises increasing the volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

13. The electronic device of claim 1, wherein obtaining a determination of a user intent corresponding to the user utterance further comprises:

obtaining a determination of a user sentiment associated with the user utterances, wherein the user intent is determined based on the determined user sentiment.

14. The electronic device of claim 1, further comprising:

in response to obtaining a determination that the user intent does not comprise a request to adjust a state or a setting of an application on the electronic device, obtaining a determination of whether the user intent is one of a plurality of predetermined request types; and

in response to obtaining a determination that the user intent is one of a plurality of predetermined request types;

obtaining results that at least partially satisfy the user intent; and

displaying, on the display unit, the results in text form.

15. The electronic device of claim 14, wherein the results in text form are overlaid on the displayed content.

16. The electronic device of claim 14, wherein the displayed content comprises media content playing on the electronic device, and wherein the media content continues to play while the results in text form are displayed.

17. The electronic device of claim 14, further comprising:

in response to obtaining a determination that the user intent is not one of a plurality of predetermined request types;

obtaining second results that at least partially satisfy the user intent;

determining whether the displayed content comprises media content playing on the electronic device; and

in accordance with a determination that the displayed content comprises media content;

determining whether the media content can be paused; and

in accordance with a determination that the media content cannot be paused, displaying on the display unit a second user interface with a portion of the second results, wherein a display area occupied by the second user interface on the display unit is smaller than a display area occupied by the media content on the display unit.

18. The electronic device of claim 17, wherein the user intent is not a media search query, and wherein the second results include one or more media items having media content that at least partially satisfies the user intent.

19. The electronic device of claim 17, wherein the user intent is a media search query and the second results comprise a plurality of media items corresponding to the media search query.

20. The electronic device of claim 17, further comprising:

in accordance with a determination that the displayed content does not comprise media content playing on the electronic device, displaying on the display unit a third user interface with a portion of the second results, wherein the third user interface occupies a majority of the display area of the display unit.

21. The electronic device of claim 20, wherein:

the displayed content comprises the third user interface with previous results related to a previous user request received prior to detecting the user input; and

in accordance with a determination that the displayed content does not comprise media content playing on the electronic device, display of the previous results in the third user interface is replaced with the display of the second results.

22. The electronic device of claim 17, further comprising:

in accordance with the determination that the displayed content comprises media content playing on the electronic device;

determining whether the displayed content includes the second user interface with previous results from a previous user request, wherein in accordance with a determination that the displayed content includes the second user interface with previous results from a previous user request, the previous results are replaced with the second results.

23. The electronic device of claim 17, further comprising:

in accordance with a determination that the media content can be paused;

pausing the playing of the media content on the electronic device;

displaying on the display unit the third user interface with a portion of the second results, wherein the third user interface occupies a majority of the display area of the display unit.

24. The electronic device of claim 1, further comprising:

obtaining a determination that the user intent does not correspond to one of a plurality of core competencies associated with the electronic device; and

causing a second electronic device to perform a task in furtherance of satisfying the user intent.

25. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:

displaying content on a display unit;

while displaying the content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining a determination of a user intent corresponding to the user utterance;

obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and

in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein;

the displayed content includes media content playing on the media device;

the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;

the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and

adjusting the state or the setting of the application comprises;

rewinding the media content by a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion.

26. A method for operating a digital assistant of a media system, the method comprising:

at an electronic device with one or more processors and memory;

displaying content on a display unit;

while displaying the content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining a determination of a user intent corresponding to the user utterance;

obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and

in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein;

the displayed content includes media content playing on the media device;

the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;

the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and

adjusting the state or the setting of the application comprises;

rewinding the media content by a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion.