DIGITAL ASSISTANT VOICE INPUT INTEGRATION

• US 20160171980A1
• Filed: 12/16/2014
• Published: 06/16/2016
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhou, Francis, Liddell, Derek, Yen, Cheng-Yi
• Granted Patent: US 10,147,421 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A digital assistant supported on devices such as smartphones, tablets, personal computers, game consoles, etc. includes an extensibility client that exposes an interface and service that enables third party applications to be integrated with the digital assistant so the application user experiences are rendered using the native voice of the digital assistant. Specific voice inputs associated with a given application may be registered by developers using a manifest that is loaded when the application is launched on the device so that voice inputs from the device user can be mapped by the digital assistant extensibility client to the appropriate application as input events for consumption. In typical implementations, the manifest is arranged as a declarative document that streamlines application development and provides a seamless user experience by enabling customization of third party applications to integrate the digital assistant'"'"'s voice and behaviors within the user experience of the application'"'"'s domain.

Claims

1. A method for implementing extensibility of a digital assistant operating on a device to one or more applications, comprising:

exposing an interface for receiving application-specific voice commands from manifests associated with respective ones of the applications;

configuring a user interface to receive voice commands from a device user;

mapping the voice commands received at the user interface to respective ones of the applications according to the manifests; and

forwarding the voice commands to the applications for handling in response to the mapping.

2. The method of claim 1 further including rendering user experiences supported by the applications using a voice associated with the digital assistant so that user experiences across the applications utilize one voice.

3. The method of claim 2 further including surfacing options to the device user for controlling characteristics of the one voice, the characteristics including one of language, gender associated with the one voice, or accent associated with the one voice.

4. The method of claim 1 further including using contextual data when performing the voice command mapping.

5. The method of claim 4 in which the contextual data comprises one or more of time/date, location of the user or device, language, schedule, applications installed on the device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, device type, device capabilities, or communication network type.

6. The method of claim 1 further including providing services to the applications, the services including one or more of language services, vocabulary services, voice services, or synthesized text to speech services.

7. The method of claim 6 in which the voice services are arranged to enable the applications to switch among different voices when rendering the user experiences.

8. The method of claim 6 further including receiving portions of the services from a remote service provider.

9. The method of claim 8 further including supporting the interface with an extensibility client that is configured for interaction with the remote service provider.

10. The method of claim 1 further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user'"'"'s calendar, scheduling a reminder, making a call, operating a device, playing games, making purchases, taking notes, scheduling an alarm or wake-up reminder, sending a message, checking social media for updates, scraping a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

11. A device, comprising:

one or more processors;

a user interface (UI) for interacting with a user of the device using graphics and audio; and

a memory device storing code associated with one or more applications and computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

exposing a digital assistant on the device for maintaining context-awareness for a device user by monitoring user behaviors and interactions with the device, the digital assistant further interacting with the device user using voice interactions through the UI,

listening for voice commands from the device user,

using the context-awareness to deliver a voice command to an application for handling, and

enabling the one or more applications to utilize one or more voices associated with the digital assistant when rendering user experiences on the device using the delivered voice command.

12. The device of claim 11 further including enabling the one or more applications to opt in to digital assistant extensibility services surfaced by a local client operating on the device.

13. The device of claim 11 further including enabling the one or more applications to load application-specific voice commands from a manifest into a runtime environment for execution.

14. The device of claim 13 in which the manifest is populated using declarative markup.

15. The device of claim 14 further including mapping instances of the application-specific voice commands to the applications for consumption as input events.

16. The device of claim 11 in which the application parses one or more chunks of text for the digital assistant to read aloud using the digital assistant'"'"'s native voice.

17. One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computer server, perform a method comprising the steps of:

interoperating with a digital assistant extensibility client on a local device, the digital assistant extensibility client exposing an application programming interface (API) to one or more applications being executable on the device;

maintaining digital assistant extensibility services including at least one of i) language services that enable applications to use one or more different languages when rendering a user experience on the local device, ii) vocabulary services that enable applications to handle unknown words or phrases when rendering the user experience, iii) voice services that enable applications to use one or more different voices when rendering the user experience, or iv) synthesized text to speech services that enable applications to use a synthesized voice that mimics a human speaker; and

providing the digital assistant extensibility services to the one or more applications through the API exposed by the digital assistant extensibility client on the local device.

18. The one or more computer-readable memory devices of claim 17 in which the digital assistant extensibility services and digital assistant extensibility client provide a platform with user experiences that are renderable on the local device using one voice across all the applications, the one voice being natively associated with a digital assistant that is executable on the local device.

19. The one or more computer-readable memory devices of claim 17 in which the applications have application-specific voice commands written to a manifest which is loaded into a runtime environment upon application launch.

20. The one or more computer-readable memory devices of claim 19 in which the digital assistant extensibility client maps voice commands from the device user to the application-specific voice commands loaded from the manifest.