Digital assistant extensibility to third party applications

• US 10,133,613 B2
• Filed: 05/14/2015
• Issued: 11/20/2018
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Surti, Tanvi, Patten, Michael, Lyndersay, Sean, Tong, Chee Chen
• Primary Examiner(s): Sough, S.
• Assistant Examiner(s): Hoang, Phuong N
• Application Number:
  US14/712,364
• Publication Number:
  US 20160335138A1
• Time in Patent Office: 1,286 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/44526 Plug-ins; Add-ons
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72406 by software upgrading or downloading
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A digital assistant includes an extensibility client that interfaces with application extensions that are built by third-party developers so that various aspects of application user experiences, content, or features may be integrated into the digital assistant and rendered as native digital assistant experiences. Application extensions can use a variety of services provided from cloud-based and/or local sources such as language/vocabulary, user preferences, and context services that add intelligence and contextual relevance while enabling the extensions to plug in and operate seamlessly within the digital assistant context. Application extensions may also access and utilize general digital assistant functions, data structures, and libraries exposed by the services and implement application domain-specific context and behaviors using the programming features captured in the extension. Such extensibility to third party applications can broaden the scope of the database of information that the digital assistant may use to answer questions and perform actions for the user.

Claims

1. A method for implementing extensibility of a digital assistant operating on a device used by a user to one or more applications, comprising:

configuring an interface for interoperating with application-specific services exposed by application extensions associated with respective ones of the applications, wherein each application extension is instantiated locally on the device and comprises a manifest of resources that are specific to a related application, an event handler, and logic;

receiving from the respective ones of the applications a request to access digital assistant resources that describe capabilities and resources to facilitate interaction between the respective ones of the applications and the digital assistant;

loading the manifest of resources from the related application to a digital assistant extensibility client, the manifest of resources including one or more keywords specific to the related application;

registering at least the one or more keywords with the digital assistant extensibility client;

receiving input from the device user to execute an event and invoke the one or more registered keywords;

based on the received input and invocation of the one or more registered keywords, mapping the device user input to an application extension for handling;

receiving an initial application-specific service from the application extension in response to the device user input; and

after receiving the initial application-specific service, receiving a subsequent application-specific service from a different application, in which the subsequent application-specific service is based on context from the initial application-specific service.

2. The method of claim 1 further including rendering the application-specific services so that user experiences across the applications are exposed to the device user as native digital assistant user experiences and whereby the application-specific services increase a size of a database of answers available to the digital assistant.

3. The method of claim 1 further including using contextual data when performing the mapping.

4. The method of claim 3 in which the contextual data comprises one or more of time/date, location of the user or device, language, schedule, applications installed on the device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, device type, device capabilities, or communication network type.

5. The method of claim 1 further including providing extensibility services to the respective ones of the applications, the extensibility services including one or more of language services, vocabulary services, user preference services, or context services.

6. The method of claim 5 further including receiving portions of the extensibility services from a remote service provider.

7. The method of claim 1 in which the application extension further includes logic for implementing a user experience or a user interface using the digital assistant.

8. The method of claim 1 further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user'"'"'s calendar, scheduling a reminder, making a call, operating a device, playing games, making purchases, taking notes, scheduling an alarm or wake-up reminder, sending a message, checking social media for updates, scraping a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

9. The method of claim 1 in which the one or more keywords specifically identify the respective application.

10. The method of claim 1 in which the subsequent application-specific service is received through a series of interactions from the user at the digital assistant.

11. A device, comprising:

one or more processors;

a user interface (UI) for interacting with a user of the device using graphics and audio; and

a memory device storing code associated with one or more applications and computer-readable instructions which, when executed by the one or more processors, cause the device to;

expose a digital assistant on the device configured to maintain context-awareness for a device user by monitoring user behaviors and interactions with the device, the digital assistant further configured to interact with the device user using voice interactions through the UI, wherein each application extension is instantiated locally on the device and comprises a manifest of resources that are specific to a related application, an event handler, and logic,

receive from the respective one or more applications a request to access digital assistant resources that describe capabilities and resources to facilitate interaction between the respective one or more applications and the digital assistant,

load the manifest of resources from the related application to a digital assistant extensibility client, the manifest of resources including one or more keywords specific to the related application,

register at least the one or more keywords with the digital assistant extensibility client,

receive an input from the device user through the UI to execute an event and invoke the one or more registered keywords,

use the context-awareness, received input, and the invocation of the one or more registered keywords to deliver the input to an application extension for handling, the application extension being configured to deliver initial services from a related application,

operate the digital assistant to render the initial services to the device user through the UI, and

after operating the digital assistant to render the initial services associated with an initial application, operate a subsequent service using a subsequent application based on context from the initial services.

12. The device of claim 11 further including instructions to cause the device to expose one or more extensibility services to the application extension.

13. The device of claim 11 further including instructions to cause the device to enable the application extension to load application-specific resources from the manifest into a runtime environment for execution.

14. The device of claim 13 in which the application extension includes the event handler.

15. One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computer server, cause the computer server to:

interoperate with a digital assistant extensibility client on a local device, the digital assistant extensibility client exposing an application programming interface (API) to one or more application extensions being executable on the local device, each of the application extensions being configured to deliver services from respective applications into user experiences renderable by a digital assistant;

maintain digital assistant extensibility services including i) language services that enable applications to use one or more different languages when rendering a user experience on the local device, or ii) vocabulary services that enable applications to handle unknown words or phrases when rendering the user experience; and

provide the digital assistant extensibility services to the one or more application extensions through the API exposed by the digital assistant extensibility client on the local device,

wherein each application extension is instantiated locally on the local device and comprises a manifest of resources that are specific to a respective application, an event handler, and logic,

wherein each application extension is loaded into a local runtime environment on the local device to receive events according to the manifest of resources, utilize the resources, and apply the logic to render a user experience by the digital assistant, and

wherein multiple application extensions are usable by the digital assistant within a series of interactions, such that previous interactions that invoke an initial application extension are utilized to identify and use subsequent application extensions.

16. The one or more computer-readable memory devices of claim 15 in which the digital assistant extensibility services and digital assistant extensibility client provide a platform supporting user experiences that are renderable on the local device across all the applications as a native digital assistant experience.

17. The one or more computer-readable memory devices of claim 15 in which the application extensions contain application-specific resources written to the manifest of resources which is loaded into the local runtime environment.

18. The one or more computer-readable memory devices of claim 17 in which the application extensions are authored by third party developers.