Force-based interactions with digital agents

• US 10,372,412 B2
• Filed: 10/25/2016
• Issued: 08/06/2019
• Est. Priority Date: 10/25/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Klein, Christian, Capuozzo, Callil R., Elliott, Brendan D., Barth, Christopher M., Tuomi, Otso Joona Casimir, Shen, Shuo
• Primary Examiner(s): Tran, Quoc A
• Application Number:
  US15/334,153
• Publication Number:
  US 20180113672A1
• Time in Patent Office: 1,015 Days
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0414 using force sensing means to determine a position
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Embodiments relate to enabling force-based interactions with an intelligent personal assistant (IPA). A computing device capable of sensing the force exerted to input touch inputs is configured with a pressure-based filter that checks pressures of touch inputs to determine which touch inputs are to be diverted to the IPA or which are to be passed on to underlying user interface that is not related to the IPA. Touch inputs designated for the IPA based on their pressure characteristics can become part of the context of the IPA. Some IPA uses of the touch inputs include selecting graphic objects on the display, resolving exophoric phrases (e.g., “that”, “those”) as referring to such selected graphic objects, displaying transient user interface to provide information about (or actions for) the selected object, incorporating a selected object into the current context of the IPA, etc.

Claims

1. A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:

executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands;

executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface;

receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device;

evaluating the force components against a force condition, wherein

(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and

(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application based on a location of the touch input and the application responds to the provided touch input.

2. A method according to claim 1, the method further comprising:

determining that an exophoric phrase is present in a given command being handled by the IPA; and

based on the determination of the presence of the exophoric phrase in the command, executing the command with reference to the target object.

3. A method according to claim 1, wherein each touch input determined to satisfy the force condition is not passed to the application, and each touch input determined to not satisfy the force condition is not used to identify a target object by the IPA.

4. A method according to claim 1, further comprising, responding, by the IPA, to identification of the target object by the IPA displaying a user interface element, wherein content of the user interface element is selected by the IPA based on one or more features of the target object.

5. A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:

executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands;

executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface;

receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device;

evaluating the force components against a force condition, wherein

(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and

(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application and the application responds to the provided touch input, and

wherein a given touch input corresponds to two candidate objects and the method further comprises the IPA and/or the application determining which of the two candidate objects will be the target object.

6. A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:

executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands;

executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface;

receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device;

evaluating the force components against a force condition, wherein

(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and

(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application and the application responds to the provided touch input, and,

wherein the force condition is used to differentiate, among the touch inputs, between first touch inputs that are to be used to identify objects by the IPA and second touch inputs that are not to be used by the IPA to identify objects.

7. A method according to claim 6, wherein the force condition comprises at least a force threshold value and the evaluating comprises comparing the force threshold value to the force components.

8. A computing device comprising:

processing hardware;

a microphone;

a touch and force sensing display that senses touch inputs and provides respective measures that correspond to forces of the touch inputs; and

storage hardware storing instructions configured to cause the processing hardware to perform a process comprising;

executing an application comprising an object displayed by the computing device;

receiving a voice command inputted through the microphone;

receiving a first touch input corresponding to a first force measure;

determining whether the force measure satisfies a force condition;

executing first logic when the determining determines that the force measure satisfies the force condition, the first logic, when executed, associating the voice command with the object, and executing the voice command based on the association between the voice command and the object; and

executing second logic when the determining does not determine that the force measure satisfies the force condition, the second logic, when executed, providing the touch input to the application based on a location of the touch input.

9. A computing device according to claim 8, wherein the associating the voice command with the object comprises identifying a graphic object displayed by the display according to the touch input, and identifying the object based on the identifying of the graphic object.

10. A computing device according to claim 8, the process further comprising determining a display region defined by the touch input and identifying a plurality of objects according to the region.

11. A computing device according to claim 8, wherein the voice command comprises an exophoric phrase and the word “

here”

or “

there”

, the process further comprising resolving the exophoric phrase as referring to the object and resolving the word “

here”

or “

there”

according to a feature of the touch input.

12. A computing device according to claim 11, wherein the storage stores an intelligent personal assistant (IPA), and wherein the IPA receives a reference to the object and generates context according to the object, the context used by the IPA to interpret and/or execute voice commands being handled by the IPA, the IPA configured to invoke applications on the computing device to execute the voice commands.

13. A computing device according to claim 12, wherein the IPA is configured to display a user interface element that comprises graphic indications of a plurality of actions or information items determined by the IPA to be relevant based on one or more properties of the object.

14. A computing device according to claim 8, the process further comprising displaying a user interface on the display, wherein the touch input is coincident with the user interface on the display, wherein the user interface is configured to respond to touch inputs, and wherein whether the touch input is received by the user interface depends on whether the touch input satisfies a threshold force condition, the threshold force condition controlling whether touch inputs coincident with the user interface are received by the user interface.

15. A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:

executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device;

receiving a touch input comprised of a pressure component and a location relative to the display, the pressure component corresponding to a pressure value sensed by the touch input device while sensing input points of the touch input; and

determining whether or not to activate the IPA by determining whether the pressure component of the touch input satisfies a pressure condition, wherein the instructions are configured to;

respond to a determination that the pressure component of the touch input satisfies the pressure condition by activating the IPA and providing the touch input to the IPA, and

respond to a determination that the pressure component of the touch input does not satisfy the pressure condition by, instead of activating the IPA and providing the touch input to the IPA, providing the touch input to an application based on the application being associated with the location of the touch input.

16. A method according to claim 15, wherein the IPA interprets the command by resolving an exophoric phrase in the command.

17. A method according to claim 16, wherein the command specifies an action or query to be invoked by the IPA, and wherein the IPA invokes the action on the object or queries for information about the object.

18. A method according to claim 15, wherein the object comprises a file system object, an object managed by an application installed on the computing device, or a link to a network resource.

19. A method according to claim 15, wherein the activating the IPA comprises causing the IPA to begin analyzing audio input received through a microphone of the computing device.

20. A method according to claim 15, wherein the method is performed by an operating system of the computing device.