Connected accessory for a voice-controlled device

• US 10,468,027 B1
• Filed: 11/12/2018
• Issued: 11/05/2019
• Est. Priority Date: 11/11/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Deller, Derick, Naik, Apoorv, Adams, Zoe, Appleman, Aslan, Cornelius, Link, Klein, Pete
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Nguyen, Timothy
• Application Number:
  US16/188,204
• Time in Patent Office: 358 Days
• Field of Search: 704235
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/047 Architecture of speech synthesisers
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 User profiles

Abstract

Coordinated operation of a voice-controlled device and an accessory device in an environment is described. A remote system processes audio data it receives from the voice-controlled device in the environment to identify a first intent associated with a first domain, a second intent associated with a second domain, and a named entity associated with the audio data. The remote system sends, to the voice-controlled device, first information for accessing main content associated with the named entity, and a first instruction corresponding to the first intent. The remote system also sends, to the accessory device, second information for accessing control information or supplemental content associated with the main content, and a second instruction corresponding to the second intent. The first and second instructions, when processed by the devices in the environment, cause coordinated operation of the voice-controlled device and the accessory device.

Claims

1. A system comprising:

one or more processors; and

non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, from a first device, audio data representing sound captured by the first device;

generating, by performing automatic speech processing on the audio data, first intent data associated with a first domain of domains of a natural language understanding system;

generating, based at least in part on the audio data, second intent data associated with a second domain of the domains;

determining control data based at least in part on the second intent data, the control data configured to control operations performed by a second device associated with the first device;

sending, to the first device and based at least in part on the first intent data, a first command causing the first device to output content during a period of time; and

sending, to the second device and based at least in part on the second intent data, a second command causing the second device to process the control data during the period of time.

2. The system of claim 1, wherein the control data causes operation of a movable member of the second device in a synchronized manner with output of audio corresponding to the content.

3. The system of claim 1, wherein the control data causes a lighting element of the second device to emit light in a synchronized manner with output of audio corresponding to the content.

4. The system of claim 1, wherein the control data causes display of images on a screen of the second device in a synchronized manner with output of audio corresponding to the content.

5. The system of claim 1, the operations further comprising:

storing the control data in association with a device identifier associated with the second device;

receiving, from the second device, request data indicating a request for the control data, the request data including an indication of the device identifier; and

wherein sending the second command to the second device comprises sending the second command to the second device based at least in part on the request data.

6. The system of claim 1, wherein the control data comprises dance data, and the operations further comprise:

identifying beat data associated with the audio data;

identifying timestamp data associated with the beat data;

identifying movement parameters of a moveable member of the second device; and

generating the dance data based at least in part on an alignment of the movement parameters with the beat data and the timestamp data.

7. The system of claim 1, the operations further comprising:

determining a number of beats per unit time based at least in part on beat data obtained from the content;

determining that the number of the beats per the unit time is equal to or greater than a predefined threshold number of beats; and

selecting an intensity level of intensity levels for operating a component of the second device based at least in part on the number of the beats per the unit time being equal to or greater than the predefined threshold number of beats.

8. The system of claim 1, wherein:

the first device is voice controllable; and

the second device is an accessory device communicatively coupled to the first device.

9. The system of claim 1, the operations further comprising:

identifying a first storage location associated with the content;

identifying a second storage location associated with the control data;

wherein the first command includes a first location identifier indicating that the content is stored in association with the first storage location; and

wherein the second command includes a second location identifier indicating that the control data is stored in association with the second storage location.

10. The system of claim 1, the operations further comprising sending, to the second device, synchronization data configured to synchronize the output of the content by the first device and processing of the control data by the second device.

11. A method comprising:

receiving, from a first device, audio data representing a user utterance;

generating, based at least in part on the audio data, first intent data associated with a first domain of domains of a speech processing system;

generating, based at least in part on the audio data, second intent data associated with a second domain of the domains;

generating control data based at least in part on the second intent data, the control data configured to control operations performed by a second device associated with the first device;

sending, to the first device and based at least in part on the first intent data, a first command causing the first device to output content during a period of time; and

sending, to the second device and based at least in part on the second intent data, a second command causing the second device to access the control data during the period of time.

12. The method of claim 11, further comprising causing operation of a movable member of the second device in a synchronized manner with output of audio corresponding to the content.

13. The method of claim 11, further comprising causing a lighting element of the second device to emit light in a synchronized manner with output of audio corresponding to the content.

14. The method of claim 11, further comprising causing display of images on a screen of the second device in a synchronized manner with output of audio corresponding to the content.

15. The method of claim 11, further comprising:

storing the control data in association with a device identifier associated with the second device;

receiving, from the second device, request data indicating a request for the control data, the request data including an indication of the device identifier; and

wherein sending the second command to the second device comprises sending the second command to the second device based at least in part on the request data.

16. The method of claim 11, wherein the control data comprises dance data, and the method further comprises:

identifying beat data associated with the content;

identifying timestamp data associated with the beat data;

identifying movement parameters of a moveable member of the second device; and

generating the dance data based at least in part on an alignment of the movement parameters with the beat data and the timestamp data.

17. The method of claim 11, further comprising storing, in a database, the control data in association with an account identifier associated with the first device and the second device.

18. The method of claim 11, further comprising:

identifying a first storage location associated with the content;

identifying a second storage location associated with the control data;

wherein the first command includes a first location identifier indicating that the content is stored in association with the first storage location; and

wherein the second command includes a second location identifier indicating that the control data is stored in association with the second storage location.

19. The method of claim 11, further comprising sending, to the second device, synchronization data configured to synchronize the output of the content by the first device and processing of the control data by the second device.

20. The method of claim 11, further comprising:

determining a device identifier associated with the second device;

determining that the device identifier is associated with a user account associated with the first device; and

wherein sending the second command to the second device comprises sending the second command to the second device based at least in part on determining that the device identifier is associated with the user account.