Intelligent automated assistant in a media environment

• US 10,331,312 B2
• Filed: 12/08/2015
• Issued: 06/25/2019
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Napolitano, Lia T., Hwang, Grace H., Penha, Henrique D., Shaw, Jeremiah D., Fino, Jorge S.
• Primary Examiner(s): Bycer, Eric J.
• Application Number:
  US14/963,094
• Publication Number:
  US 20170068423A1
• Time in Patent Office: 1,295 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/438 Presentation of query results
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04N 21/41265 : having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 : characterized by hardware details
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/47 : End-user applications
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739

Abstract

Systems and processes are disclosed for operating a digital assistant in a media environment. In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one approach, a plurality of exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The plurality of exemplary natural language requests can be contextually-related to the displayed content. In another approach, a user request can be received in response to detecting a user input of a second input type. A task that at least partially satisfies the user request can be performed. The performed task can depend on the nature of the user request and the content being displayed by the media device. In particular, the user request can be satisfied while reducing disruption to user consumption of media content.

Claims

1. A non-transitory computer-readable medium storing instructions for operating a digital assistant of a media system, the instructions, when executed by one or more processors of an electronic device, cause the one or more processors to:

display content on a display;

detect a first user input corresponding to an affordance on a secondary electronic device;

determine whether the first user input corresponds to a first input type, said first input type comprising a non-verbal input type; and

in accordance with a determination that the first user input corresponds to a first input type;

display, on the display, a plurality of exemplary natural language requests, the plurality of exemplary natural language requests being contextually-related to the displayed content; and

in response to a second user input corresponding to the affordance on the secondary electronic device;

sample audio data; and

determine whether the sampled audio data includes a user request.

2. The non-transitory computer-readable medium of claim 1, wherein the plurality of exemplary natural language requests are displayed on the display via a first user interface, and wherein the first user interface is overlaid on the displayed content.

3. The non-transitory computer-readable medium of claim 1, wherein the displayed content comprises media content, and wherein the media content continues to play while displaying the plurality of exemplary natural language requests.

4. The non-transitory computer-readable medium of claim 1, wherein each of the plurality of exemplary natural language requests is displayed separately in a predetermined sequence and at different times.

5. The non-transitory computer-readable medium of claim 1, wherein displaying the plurality of exemplary natural language requests comprises:

displaying a plurality of lists of exemplary natural language requests, wherein each list is displayed at a different time and on a rotating basis.

6. The non-transitory computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the first user input does not correspond to a first input type;

determine whether the first user input corresponds to a second input type; and

in accordance with a determination that the first user input corresponds to a second input type;

sample second audio data;

determine whether the sampled second audio data contains a second user request; and

in accordance with a determination that the sampled second audio data contains a second user request, perform a task that at least partially satisfies the second user request.

7. The non-transitory computer-readable medium of claim 6, wherein the displayed content comprises media content, and wherein the media content continues to play on the electronic device while sampling the second audio data and while performing the task.

8. The non-transitory computer-readable medium of claim 7, wherein the instructions further cause the one or more processors to:

output audio associated with the media content; and

in accordance with a determination that the first user input corresponds to a second input type, reduce an amplitude of the audio.

9. The non-transitory computer-readable medium of claim 6, wherein the second audio data is sampled while detecting the first user input.

10. The non-transitory computer-readable medium of claim 6, wherein audio associated with the displayed content is outputted based on an audio signal from the electronic device, and wherein the instructions further cause the one or more processors to:

remove background noise in the sampled second audio data using the audio signal.

11. The non-transitory computer-readable medium of claim 6, wherein:

the displayed content includes media content playing on the electronic device;

the second user request comprises a natural language expression indicating that a user did not hear a portion of audio associated with the media content; and

performing the task includes;

rewinding the media content by a predetermined amount to a previous portion of the media content; and

restarting play of the media content from the previous portion of the media content.

12. The non-transitory computer-readable medium of claim 11, wherein performing the task further includes:

turning on closed captioning prior to restarting play of the media content from the previous portion.

13. The non-transitory computer-readable medium of claim 11, wherein performing the task further includes increasing a volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

14. The non-transitory computer-readable medium of claim 6, wherein the performed task comprises:

obtaining results that at least partially satisfy the second user request; and

displaying a second user interface on the display, the second user interface including a portion of the results, wherein at least a portion of the content continues to be displayed while the second user interface is displayed, and wherein a display area of the second user interface on the display is smaller than a display area of the at least a portion of the content on the display.

15. The non-transitory computer-readable medium of claim 14, wherein the portion of the results includes one or more media items, and wherein the instructions further cause the one or more processors to:

receive a selection of a media item of the one or more media items via the second user interface; and

display media content associated with the selected media item on the display.

16. The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the one or more processors to:

while displaying the second user interface, detect a third user input; and

in response to detecting the third user input, cease to display the second user interface.

17. The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the one or more processors to:

while displaying the second user interface, detect a fourth user input; and

in response to detecting the fourth user input, replace display of the second user interface with display of a third user interface on the display, the third user interface including at least the portion of the results, wherein the third user interface occupies at least a majority of a display area of the display.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the one or more processors to:

in response to detecting the fourth user input;

obtain second results that are different from the results, wherein the second results at least partially satisfy the second user request, and wherein the third user interface includes at least a portion of the second results.

19. The non-transitory computer-readable medium of claim 18, wherein a focus of the second user interface is on an item of the portion of the results while the fourth user input is detected, and wherein the second results are contextually-related to the item.

20. The non-transitory computer-readable medium of claim 17, wherein the displayed content comprises media content, and wherein playing of the media content on the electronic device is paused in response to detecting the fourth user input.

21. The non-transitory computer-readable medium of claim 17, wherein the at least the portion of the results includes one or more media items, and wherein the instructions further cause the one or more processors to:

receive a selection of a media item of the one or more media items via the third user interface; and

display media content associated with the media item on the display.

22. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the one or more processors to:

while displaying the third user interface, detect a fifth user input associated with a direction on the display; and

in response to detecting the fifth user input;

switch a focus of the third user interface from a first item to a second item on the third user interface, the second item being positioned in the direction relative to the first item.

23. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the one or more processors to:

while displaying the third user interface, detect a sixth user input; and

in response to detecting the sixth user input;

display a search field; and

display a virtual keyboard interface on the display, wherein input received via the virtual keyboard interface causes text entry in the search field.

24. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the one or more processors to:

while displaying the third user interface, detect a seventh user input; and

in response to detecting the seventh user input;

sample third audio data, the third audio data containing a third user request;

determine whether the third user request is a request to refine the results of the second user request; and

in accordance with a determination that the third user request is a request to refine the results of the second user request, display a subset of the results via the third user interface.

25. The non-transitory computer-readable medium of claim 24, wherein the instructions further cause the one or more processors to:

in accordance with a determination that the third user request is not a request to refine the results of the second user request;

obtain third results that at least partially satisfy the third user request; and

display a portion of the third results via the third user interface.

26. The non-transitory computer-readable medium of claim 24, wherein the instructions further cause the one or more processors to:

obtain fourth results that at least partially satisfy the second user request or the third user request; and

display a portion of the fourth results via the third user interface.

27. The non-transitory computer-readable medium of claim 26, wherein a focus of the third user interface is on one or more items of the third user interface while the seventh user input is detected, and wherein the fourth results are contextually-related to the one or more items.

28. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the one or more processors to:

while displaying the third user interface, detect an eighth user input; and

in response to detecting the eighth user input, cease to display the third user interface, wherein the displayed content is media content, wherein playing of the media content on the electronic device is paused in response to detecting the fourth user input, and wherein playing of media content on the electronic device is resumed in response to detecting the eighth user input.

29. A method for operating a digital assistant of a media system, the method comprising:

at an electronic device with one or more processors and memory;

displaying content on a display;

detecting a first user input corresponding to an affordance on a secondary electronic device;

determining whether the first user input corresponds to a first input type, said first input type comprising a non-verbal input type; and

in accordance with a determination that the first user input corresponds to a first input type;

displaying, on the display unit, a plurality of exemplary natural language requests, the plurality of exemplary natural language requests being contextually-related to the displayed content; and

in response to a second user input corresponding to the affordance on the secondary electronic device;

sample audio data; and

determine whether the sampled audio data includes a user request.

30. The method of claim 29, wherein said first input type being a non-verbal input type comprises pressing a button.

31. The method of claim 29, wherein displaying, on the display, a plurality of exemplary natural language requests, the plurality of exemplary natural language requests being contextually-related to the displayed content comprises displaying, on the display at least one exemplary natural language request that comprises a contextually ambiguous term.

32. The method of claim 29, further comprising in accordance with a determination that the first user input corresponds to a first input type:

concurrently displaying, on the display, a passive visual indicator together with the plurality of exemplary natural language requests.

33. A system for operating a digital assistant, the system comprising:

one or more processors; and

memory storing instructions, which when executed by the one or more processors, cause the one or more processors to;

display content on a display;

detect a first user input corresponding to an affordance on a secondary electronic device;

determine whether the first user input corresponds to a first input type, said first input type comprising a non-verbal input type; and

in accordance with a determination that the first user input corresponds to a first input type;

display, on the display, a plurality of exemplary natural language requests, the plurality of exemplary natural language requests being contextually-related to the displayed content; and

in response to a second user input corresponding to the affordance on the secondary electronic device;

sample audio data; and

determine whether the sampled audio data includes a user request.