INTELLIGENT AUTOMATED ASSISTANT FOR TV USER INTERACTIONS

• US 20160212488A1
• Filed: 03/30/2016
• Published: 07/21/2016
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: OS, Marcel Van, SADDLER, Harry J., NAPOLITANO, Lia T., RUSSELL, Jonathan H., LISTER, Patrick M., DASARI, Rohit
• Granted Patent: US 9,668,024 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09C 1/00 Apparatus or methods whereby a given sequence of signs, e.g. an intelligible text, is transformed into an unintelligible sequence of signs by transposing the signs or groups of signs or by replacing them by others according to a predetermined system cryptographic typewriters G09C3/00
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 21/2541 Rights Management protecting software against unauthorised usage in a vending or licensing environment G06F21/10; security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4438 : Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/4532 : involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4627 : Rights management associated to the content security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/47214 : for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/4722 : for requesting additional data associated with the content
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4826 : using recommendation lists, e.g. of programs or channels sorted out according to their score
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/4882 : for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/858 : Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot

Abstract

Systems and processes are disclosed for controlling television user interactions using a virtual assistant. A virtual assistant can interact with a television set-top box to control content shown on a television. Speech input for the virtual assistant can be received from a device with a microphone. User intent can be determined from the speech input, and the virtual assistant can execute tasks according to the user'"'"'s intent, including causing playback of media on the television. Virtual assistant interactions can be shown on the television in interfaces that expand or contract to occupy a minimal amount of space while conveying desired information. Multiple devices associated with multiple displays can be used to determine user intent from speech input as well as to convey information to users. In some examples, virtual assistant query suggestions can be provided to the user based on media content shown on a display.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device having a processor, memory, and a first display, cause the device to:

receive speech input from a user;

determine a user intent of the speech input based on content displayed on the first display;

determine media content based on the user intent; and

play the media content on a second device associated with a second display.

2. The computer-readable storage medium of claim 1, wherein the content displayed on the first display comprises an application interface.

3. The computer-readable storage medium of claim 2, wherein the speech input comprises a request to display media associated with the application interface.

4. The computer-readable storage medium of claim 3, wherein the application interface comprises a television program listing, and the media comprises a television program in the television program listing.

5. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

identify the user based on the speech input; and

determine the user intent of the speech input based on data associated with the identified user.

6. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

determine whether the user is an authorized user based on the speech input;

wherein the media content is played on the second device in response to a determination that the user is an authorized user.

7. The computer-readable storage medium of claim 6, wherein determining whether the user is an authorized user comprises analyzing the speech input using voice recognition.

8. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

in response to determining that the user intent comprises a request for information, display information associated with the media content on the first display of the device; and

in response to determining that the user intent comprises a request to play the media content, play the media content on the second device.

9. The computer-readable storage medium of claim 1, wherein the speech input comprises a request to play content on the second device;

and

wherein the media content is played on the second device in response to the request to play content on the second device.

10. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

determine whether the determined media content should be displayed on the first display or the second display based on a media format, a user preference, or a default setting;

wherein the media content is displayed on the second display in response to a determination that the determined media content should be displayed on the second display; and

wherein the media content is displayed on the first display in response to a determination that the determined media content should be displayed on the first display.

11. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

determine a proximity of each of two or more devices, including the second device and a third device;

wherein the media content is played on the second device associated with the second display based on the proximity of the second device relative to the proximity of the third device.

12. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

display a list of display devices, including the second device associated with the second display; and

receive a selection of the second device in the list of display devices;

wherein the media content is displayed on the second display in response to receiving the selection of the second device.

13. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

determine whether headphones are attached to the device;

in response to a determination that headphones are attached to the device, display the media content on the first display; and

in response to a determination that headphones are not attached to the device, display the media content on the second display.

14. The computer-readable storage medium of claim 1, wherein the instructions further cause the device to:

determine alternative media content for display based on one or more of a user preference, a show popularity, and a status of a live sporting event; and

display a notification comprising the determined alternative media content.

15. A method for controlling television interactions using a virtual assistant, the method comprising:

at an electronic device having a first display;

receiving speech input from a user;

determining a user intent of the speech input based on content displayed on the first display;

determining media content based on the user intent; and

playing the media content on a second device associated with a second display.

16. The method of claim 15, further comprising:

identifying the user based on the speech input; and

determining the user intent of the speech input based on data associated with the identified user.

17. The method of claim 15, further comprising:

in response to determining that the user intent comprises a request for information, displaying information associated with the media content on the first display of the device; and

in response to determining that the user intent comprises a request to play the media content, playing the media content on the second device.

18. The method of claim 15, further comprising:

determining whether the determined media content should be displayed on the first display or the second display based on a media format, a user preference, or a default setting;

wherein the media content is displayed on the second display in response to a determination that the determined media content should be displayed on the second display; and

wherein the media content is displayed on the first display in response to a determination that the determined media content should be displayed on the first display.

19. The method of claim 15, further comprising:

displaying a list of display devices, including the second device associated with the second display; and

receiving a selection of the second device in the list of display devices;

wherein the media content is displayed on the second display in response to receiving the selection of the second device.

20. The method of claim 15, further comprising:

determining alternative media content for display based on one or more of a user preference, a show popularity, and a status of a live sporting event; and

displaying a notification comprising the determined alternative media content.

21. An electronic device comprising:

a first display;

one or more processors; and

memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for;

receiving speech input from a user;

determining a user intent of the speech input based on content displayed on the first display;

determining media content based on the user intent; and

playing the media content on a second device associated with a second display.

22. The device of claim 21, wherein the one or more programs further include instructions for:

identifying the user based on the speech input; and

determining the user intent of the speech input based on data associated with the identified user.

23. The device of claim 21, wherein the one or more programs further include instructions for:

in response to determining that the user intent comprises a request for information, displaying information associated with the media content on the first display of the device; and

in response to determining that the user intent comprises a request to play the media content, playing the media content on the second device.

24. The device of claim 21, wherein the one or more programs further include instructions for:

determining whether the determined media content should be displayed on the first display or the second display based on a media format, a user preference, or a default setting;

wherein the media content is displayed on the second display in response to a determination that the determined media content should be displayed on the second display; and

wherein the media content is displayed on the first display in response to a determination that the determined media content should be displayed on the first display.

25. The device of claim 21, wherein the one or more programs further include instructions for:

displaying a list of display devices, including the second device associated with the second display; and

receiving a selection of the second device in the list of display devices;

wherein the media content is displayed on the second display in response to receiving the selection of the second device.

26. The device of claim 21, wherein the one or more programs further include instructions for:

determining alternative media content for display based on one or more of a user preference, a show popularity, and a status of a live sporting event; and

displaying a notification comprising the determined alternative media content.