Intelligent automated assistant in a media environment

• US 10,379,715 B2
• Filed: 06/20/2017
• Issued: 08/13/2019
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Napolitano, Lia T., Hwang, Grace H., Penha, Henrique D., Shaw, Jeremiah D., Fino, Jorge S.
• Primary Examiner(s): Bycer, Eric J.
• Application Number:
  US15/627,869
• Publication Number:
  US 20170285915A1
• Time in Patent Office: 784 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/438 Presentation of query results
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04N 21/41265 : having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 : characterized by hardware details
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/47 : End-user applications
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739

Abstract

Systems and processes are disclosed for operating a digital assistant in a media environment. In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one approach, a plurality of exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The plurality of exemplary natural language requests can be contextually-related to the displayed content. In another approach, a user request can be received in response to detecting a user input of a second input type. A task that at least partially satisfies the user request can be performed. The performed task can depend on the nature of the user request and the content being displayed by the media device. In particular, the user request can be satisfied while reducing disruption to user consumption of media content.

Claims

1. An electronic device for operating a digital assistant of a media system comprising:

memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for;

displaying media content on a display unit;

while displaying the media content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining, based on prosody information, a determination of a user sentiment associated with the user utterance;

obtaining a determination of a user intent corresponding to the user utterance, wherein the user intent is determined based on the determined user sentiment;

in accordance with the determined user sentiment corresponding to a first user sentiment, performing a first task flow corresponding to the determined user intent, wherein performing the first task flow includes;

skipping a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion; and

in accordance with the determined user sentiment corresponding to a second user sentiment different from the first user sentiment, performing a second task flow corresponding to the determined user intent, wherein the second task flow is different from the first task flow.

2. The electronic device of claim 1, wherein the displayed media content includes a user interface with a media item, wherein the user utterance does not explicitly define the particular media item to be played, and the one or more programs further include instructions for:

determining whether a focus of the user interface is on the media item; and

in accordance with a determination that a focus of the user interface is on the media item, identifying the media item as the particular media item to be played.

3. The electronic device of claim 1, wherein the determined user intent comprises a request to view a user interface of the media device.

4. The electronic device of claim 1, wherein the determined user intent comprises a request to pause the playing of the media content on the electronic device.

5. The electronic device of claim 1, wherein the determined user intent comprises a request to turn-on or turn-off subtitles of the media content.

6. The electronic device of claim 1, wherein the one or more programs further include instructions for:

turning on closed captioning prior to restarting play of the media content from the previous portion.

7. The electronic device of claim 1, wherein:

the determined user intent further comprises a request to increase a volume of the audio associated with the media content; and

performing the first task flow includes increasing the volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

8. The electronic device of claim 1, wherein the one or more programs further include instructions for:

obtaining a determination that the user intent does not correspond to a task associated with a capability of the digital assistant;

determining a task flow for satisfying the user intent;

transmitting the task flow to a second electronic device; and

causing the second electronic device to perform a task in accordance with the task flow.

9. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:

displaying media content on a display unit;

while displaying the media content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining, based on prosody information, a determination of a user sentiment associated with the user utterance;

obtaining a determination of a user intent corresponding to the user utterance, wherein the user intent is determined based on the determined user sentiment;

in accordance with the determined user sentiment corresponding to a first user sentiment, performing a first task flow corresponding to the determined user intent, wherein performing the first task flow includes;

skipping a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion; and

in accordance with the determined user sentiment corresponding to a second user sentiment different from the first user sentiment, performing a second task flow corresponding to the determined user intent, wherein the second task flow is different from the first task flow.

11. The computer-readable storage medium of claim 9, wherein the displayed media content includes a user interface with a media item, wherein the user utterance does not explicitly define the particular media item to be played, and the one or more programs further include instructions for:

determining whether a focus of the user interface is on the media item; and

in accordance with a determination that a focus of the user interface is on the media item, identifying the media item as the particular media item to be played.

12. The computer-readable storage medium of claim 9, wherein the determined user intent comprises a request to view a user interface of the media device.

13. The computer-readable storage medium of claim 9, wherein the determined user intent comprises a request to pause the playing of the media content on the electronic device.

14. The computer-readable storage medium of claim 9, wherein the determined user intent comprises a request to turn-on or turn-off subtitles of the media content.

15. The computer-readable storage medium of claim 9, wherein the one or more programs further include instructions for:

turning on closed captioning prior to restarting play of the media content from the previous portion.

16. The computer-readable storage medium of claim 9, wherein:

the determined user intent further comprises a request to increase a volume of the audio associated with the media content; and

performing the first task flow includes increasing the volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

17. The computer-readable storage medium of claim 9, wherein the one or more programs further include instructions for:

obtaining a determination that the user intent does not correspond to a task associated with a capability of the digital assistant;

determining a task flow for satisfying the user intent;

transmitting the task flow to a second electronic device; and

causing the second electronic device to perform a task in accordance with the task flow.

10. A method for operating a digital assistant of a media system, the method comprising:

At an electronic device with one or more processor and memory;

displaying media content on a display unit;

while displaying the media content, detecting a user input;

in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;

obtaining, based on prosody information, a determination of a user sentiment associated with the user utterance;

obtaining a determination of a user intent corresponding to the user utterance, wherein the user intent is determined based on the determined user sentiment;

in accordance with the determined user sentiment corresponding to a first user sentiment, performing a first task flow corresponding to the determined user intent, wherein performing the first task flow includes;

skipping a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion; and

in accordance with the determined user sentiment corresponding to a second user sentiment different from the first user sentiment, performing a second task flow corresponding to the determined user intent, wherein the second task flow is different from the first task flow.

18. The method of claim 10, wherein the displayed media content includes a user interface with a media item, and wherein the user utterance does not explicitly define the particular media item to be played, further comprising:

determining whether a focus of the user interface is on the media item; and

in accordance with a determination that a focus of the user interface is on the media item, identifying the media item as the particular media item to be played.

19. The method of claim 10, wherein the determined user intent comprises a request to view a user interface of the media device.

20. The method of claim 10, wherein the determined user intent comprises a request to pause the playing of the media content on the electronic device.

21. The method of claim 10, wherein the determined user intent comprises a request to turn-on or turn-off subtitles of the media content.

22. The method of claim 10, further comprising:

turning on closed captioning prior to restarting play of the media content from the previous portion.

23. The method of claim 10, wherein:

the determined user intent further comprises a request to increase a volume of the audio associated with the media content; and

performing the first task flow includes increasing the volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

24. The method of claim 10, further comprising:

obtaining a determination that the user intent does not correspond to a task associated with a capability of the digital assistant;

determining a task flow for satisfying the user intent;

transmitting the task flow to a second electronic device; and

causing the second electronic device to perform a task in accordance with the task flow.