Connecting assistant device to devices

• US 10,353,480 B2
• Filed: 05/24/2017
• Issued: 07/16/2019
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Roman, Manuel, Segal, Mara Clair, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Wozniak, James S
• Application Number:
  US15/604,325
• Publication Number:
  US 20180301148A1
• Time in Patent Office: 783 Days
• Field of Search: 704270, 7042701, 704271, 704275
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

The present disclosure contemplates a variety of improved methods and systems for enabling set up of a variety of disparate IoT devices coupled to the framework and/or ambient operating system of an assistant device. The described solution includes a voice-driven assistant device setup process. An assistant device can determine the IoT devices in its physical environment and provide setup instructions to the user. The setup including determining voice activatable commands, device information and the adapter to allow the assistant device to operate the one or more IoT devices in response to user instructions.

Claims

1. An assistant device, comprising:

a display screen;

a microphone;

a speaker;

a camera;

one or more processors; and

memory storing instructions, wherein the one or more processors are configured to execute instructions such that the one or more processors and memory are configured to;

receive a first image data depicting one or more devices in an environment of the assistant device from the camera;

identify the one or more devices depicted in the first image data using a visual recognition algorithm;

determine that the one or more devices in the received first image data were not previously connected to the assistant device;

determine device information representing identifying information associated with the one or more devices not previously connected to the assistant device;

determine a device type of the one or more devices based on the device information using one or more of local resources and/or one or more of cloud servers;

determine first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;

provide the first audio installation instructions via the speaker;

receive a second image data depicting a user performing the first audio installation instructions;

determine characteristics of the user performing the first audio installation instructions based on the second image data;

determine second audio installation instructions for the one or more devices based on the device type and the characteristics of the user, the second audio installation instructions providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions;

provide the second audio installation instructions via the speaker;

receive a user response via the microphone and the camera;

determine descriptive information associated with the one or more devices based on the user response;

set up the assistant device to operate the one or more devices, the set up including determining one or more of voice activatable commands and an adapter associated with the one or more devices;

receive a user instruction, wherein speech recognition of the user instruction includes one or more voice activatable commands and one or more descriptive information;

identify the one or more devices associated with the descriptive information; and

transmit a request to the one or more devices to perform an action associated with the one or more voice activatable commands.

2. The assistant device of claim 1, wherein determining the device type includes analyzing an image frame of the image using the visual recognition algorithm.

3. The assistant device of claim 2, comprising determination a location of the one or more devices by analyzing the image frame of the image.

4. The assistant device of claim 1, wherein determining characteristics of the user performing the first audio installation instructions includes determining physical movements of the user while performing the first audio installation instructions.

5. The assistant device of claim 1, wherein providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions includes providing the second audio installation instructions at a slower speed than the first audio installation instructions.

6. A method, comprising:

receiving a first image data depicting one or more devices in an environment of an assistant device from a camera;

identifying the one or more devices depicted in the first image data using a visual recognition algorithm;

determining that the one or more devices in the received first image data were not previously connected to the assistant device;

determining device information representing identifying information associated with the one or more devices not previously connected to the assistant device;

determining a device type of the one or more devices based on the device information using one or more of local resources or one or more of cloud servers;

determining first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;

providing the first audio installation instructions via a speaker;

receiving a second image data depicting a user performing the first audio installation instructions;

determining characteristics of the user performing the first audio installation instructions based on the second image data;

determining second audio installation instructions for the one or more devices based on the device type and the characteristics of the user, the second audio installation instructions providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions;

providing the second audio installation instructions via the speaker;

receiving a user response via a microphone and the camera;

determining descriptive information associated with the one or more devices based on the user response;

setting up the assistant device to operate the one or more devices, the set up including determining one or more of voice activatable commands and an adapter associated with the one or more devices;

receiving a user instruction, wherein speech recognition of the user instruction includes one or more voice activatable commands and one or more descriptive information;

identifying the one or more devices associated with the descriptive information; and

transmitting a request to the one or more devices to perform an action associated with the one or more voice activatable commands.

7. The method of claim 6, wherein determining the device type includes analyzing an image frame of the image using the visual recognition algorithm.

8. The method of claim 7, comprising determination a location of the one or more devices by analyzing the image frame of the image.

9. The method of claim 6, wherein determining characteristics of the user performing the first audio installation instructions includes determining physical movements of the user while performing the first audio installation instructions.

10. The method of claim 6, wherein providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions includes providing the second audio installation instructions at a slower speed than the first audio installation instructions.

11. A computer program product including one or more non-transitory computer-readable media storing computer program instructions, execution of which by a processor causes a processing system to perform operations comprising:

receiving a first image data depicting one or more devices in an environment of an assistant device from a camera;

identifying the one or more devices depicted in the first image data using a visual recognition algorithm;

determining that the one or more devices in the received first image data were not previously connected to the assistant device;

determining device information representing identifying information associated with the one or more devices not previously connected to the assistant device;

determining a device type of the one or more devices based on the device information using one or more of local resources or one or more of cloud servers;

determining first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;

providing the first audio installation instructions via a speaker;

receiving a second image data depicting a user performing the first audio installation instructions;

determining characteristics of the user performing the first audio installation instructions based on the second image data;

determining second audio installation instructions for the one or more devices based on the device type and the characteristics of the user, the second audio installation instructions providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions;

providing the second audio installation instructions via the speaker;

receiving a user response via a microphone and the camera;

determining descriptive information associated with the one or more devices based on the user response;

setting up the assistant device to operate the one or more devices, the set up including determining one or more of voice activatable commands and an adapter associated with the one or more devices;

receiving a user instruction, wherein speech recognition of the user instruction includes one or more voice activatable commands and one or more descriptive information;

identifying the one or more devices associated with the descriptive information; and

transmitting a request to the one or more devices to perform an action associated with the one or more voice activatable commands.

12. The computer program product of claim 11, wherein determining the device type includes analyzing an image frame of the image using the visual recognition algorithm.

13. The computer program product of claim 12, comprising determination a location of the one or more devices by analyzing the image frame of the image.

14. The computer program product of claim 11, wherein determining characteristics of the user performing the first audio installation instructions includes determining physical movements of the user while performing the first audio installation instructions.

15. The computer program product of claim 11, wherein providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions includes providing the second audio installation instructions at a slower speed than the first audio installation instructions.