Voice setup instructions

• US 10,176,807 B2
• Filed: 05/24/2017
• Issued: 01/08/2019
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Roman, Manuel, Segal, Mara Clair, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Shin, Seong Ah A
• Application Number:
  US15/604,358
• Publication Number:
  US 20180301149A1
• Time in Patent Office: 594 Days
• Field of Search: 704235, 704254, 704275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/04 Time compression or expansion
• CPC Class Codes: H04L 12/2807 Exchanging configuration information on appliance services in a home automation network arrangements for maintenance or administration involving network analysis for automatically determining the actual topology of a network H04L41/12; hardware or software tools for network management using graphical user interfaces H04L41/22; address allocation H04L61/50

Abstract

A home assistant device assisting with the setup of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined to include a step requesting information related to the setup of another device. The setup of the devices can be ordered based on that determination. The setup instructions can then be provided.

Claims

1. A method for a home assistant device with artificial intelligence (AI) to facilitate setup of devices within a home, comprising:

determining, by a processor, that a first device and a second device are to be set up within the home;

determining, by the processor, first setup instructions for the first device and second setup instructions for the second device, each of the first setup instructions and the second setup instructions representing a corresponding sequence of steps to be performed to set up the corresponding device to operate within the home;

determining, by the processor, that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the information including authentication credentials to access a wireless network provided by the first device;

generating, by the processor, an order of performing the first setup instructions and the second setup instructions based on the determination that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the order indicating that the first setup instructions for the first device are to be performed before the second setup instructions for the second device;

receiving, by the processor, first image data providing content of an environment of the home assistant device;

determining, by the processor, descriptive information related to the first device in the environment based on the first image data, the descriptive information including one or more of a color of the first device, a location of the first device within the home, or a type of the first device;

providing, by the processor, first voice audio including speech content describing a first step of the first setup instructions for the user to perform, the speech content including references to the descriptive information of the first device to facilitate the setup of the first device, the first step of the first setup instructions representing setup of the authentication credentials of the wireless network provided by the first device;

receiving, by the processor, second image data providing content of the environment of the home assistant device, the content of the second image data including a depiction of a user in the environment that is manipulating the first device when performing the first step of the first setup instructions;

receiving, by the processor, first audio data providing audio corresponding to the user when performing the first step of the first setup instructions;

providing, by the processor, second voice audio including speech content describing a second step of the first setup instructions for the user to perform, the second step following the first step in the sequence of steps for the first setup instructions, the first voice audio having first characteristics, the second voice audio having second characteristics that are different than the first characteristics, the characteristics of the second voice audio based on the second image data and the first audio data, the characteristics including one or more of speed of the speech content describing the second step, accent of the speech content, or repetition of the speech content;

determining, by the processor, that the first setup instructions are completed following the second step of the first setup instructions; and

providing, by the processor, third voice audio including speech content describing a first step of the second setup instructions to set up the second device for the user to perform after the determination that the first setup instructions for the first device are completed, the speech content including a reference to the authentication credentials of the wireless network set up with the first step of the first setup instructions for the first device.

2. A method, comprising:

determining first setup instructions for a first device and second setup instructions for a second device, wherein the first setup instructions for the first device includes a first step and a second step;

determining that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the information relating to access to a resource provided by the first device;

determining, by a processor, an order of performing the first setup instructions and the second setup instructions based on the determination that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the order representing that the first setup instructions for the first device are to be performed before the second setup instructions for the second device;

receiving image data providing visual content of an environment including the first device;

determining descriptive information related to the first device in the environment based on the image data;

providing first voice audio including content describing the first setup instructions, the first setup instructions representing setup of the resource provided by the first device, wherein the content of the first voice audio references the descriptive information related to the first device, wherein providing the first voice audio also includes;

providing first audio instructions describing the first step,

receiving audio data providing content corresponding to the user when performing the first step of the first setup instructions,

receiving image data providing content depicting a user manipulating the first device when performing the first step of the first setup instructions, and

providing second audio instructions describing the second step, the first audio instructions having first characteristics, the second audio instructions having second characteristics that are different than the first characteristics, the second characteristics based on the audio data providing content corresponding to the user when performing the first step of the first setup instructions and the image data, wherein the second characteristics include one or more of;

speed of the second audio instructions describing the second step, accent of the second audio instructions, or repetition of the second audio instructions; and

providing second voice audio including content describing the second setup instructions, the second setup instructions including setting up the second device to use the resource provided by the first device.

3. The method of claim 2, wherein the resource provided by the first device is a wireless network, and the information related to setup of the first device includes authentication credentials for the wireless network.

4. The method of claim 2, wherein the descriptive information includes one or more of a color of the first device, a location of the first device within the environment, or a type of the first device.

5. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:

determine first setup instructions for a first device and second setup instructions for a second device, wherein the first setup instructions for the first device includes a first step and a second step;

determine that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the information relating to access to a resource provided by the first device;

determine an order of performing the first setup instructions and the second setup instructions based on the determination that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the order representing that the first setup instructions for the first device are to be performed before the second setup instructions for the second device;

receive image data providing visual content of an environment including the first device;

determine descriptive information related to the first device in the environment based on the image data;

provide first voice audio including content describing the first setup instructions, the first setup instructions representing setup of the resource provided by the first device, wherein the content of the first voice audio references the descriptive information related to the first device, wherein providing the first voice audio also includes;

provide first audio instructions describing the first step,

receive audio data providing content corresponding to the user when performing the first step of the first setup instructions,

receive image data providing content depicting a user manipulating the first device when performing the first step of the first setup instructions, and

provide second audio instructions describing the second step, the first audio instructions having first characteristics, the second audio instructions having second characteristics that are different than the first characteristics, the second characteristics based on the audio data providing content corresponding to the user when performing the first step of the first setup instructions and the image data, wherein the second characteristics include one or more of;

speed of the second audio instructions describing the second step, accent of the second audio instructions, or repetition of the second audio instructions; and

provide second voice audio including content describing the second setup instructions, the second setup instructions including setting up the second device to use the resource provided by the first device.

6. The computer program product of claim 5, wherein the resource provided by the first device is a wireless network, and the information related to setup of the first device includes authentication credentials for the wireless network.

7. The computer program product of claim 5, wherein the descriptive information includes one or more of a color of the first device, a location of the first device within the environment, or a type of the first device.

8. An electronic device, comprising:

memory configured to store first setup instructions for a first device and second setup instructions for a second device, wherein the first setup instructions for the first device includes a first step and a second step; and

a controller configured to access the memory to determine the first setup instructions and the second setup instructions, and configured to;

determine that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the information relating to access to a resource provided by the first device;

determine an order of performing the first setup instructions and the second setup instructions based on the determination that the second setup instructions corresponding to the second device includes a step requesting information related to setup of the first device, the order representing that the first setup instructions for the first device are to be performed before the second setup instructions for the second device;

receive image data providing visual content of an environment including the first device;

determine descriptive information related to the first device in the environment based on the image data;

provide first voice audio including content describing the first setup instructions, the first setup instructions representing setup of the resource provided by the first device, wherein the content of the first voice audio references the descriptive information related to the first device, wherein providing the first voice audio also includes;

provide first audio instructions describing the first step,

receive audio data providing content corresponding to the user when performing the first step of the first setup instructions,

receive image data providing content depicting a user manipulating the first device when performing the first step of the first setup instructions, and

provide second audio instructions describing the second step, the first audio instructions having first characteristics, the second audio instructions having second characteristics that are different than the first characteristics, the second characteristics based on the audio data providing content corresponding to the user when performing the first step of the first setup instructions and the image data, wherein the second characteristics include one or more of;

speed of the second audio instructions describing the second step, accent of the second audio instructions, or repetition of the second audio instructions; and

provide second voice audio including content describing the second setup instructions, the second setup instructions including setting up the second device to use the resource provided by the first device.

9. The electronic device of claim 8, wherein the resource provided by the first device is a wireless network, and the information related to setup of the first device includes authentication credentials for the wireless network.

10. The electronic device of claim 8, wherein the descriptive information includes one or more of a color of the first device, a location of the first device within the environment, or a type of the first device.