Scripting Cross-Device Wearable Interaction

• US 20160085366A1
• Filed: 09/21/2015
• Published: 03/24/2016
• Est. Priority Date: 09/22/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yang, Chi, Pei-Yu
• Granted Patent: US 10,073,553 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Methods and apparatus for an interaction system specifying cross-device interaction are provided. The interaction system provides application program interfaces (APIs) and a scripting environment that allow development of scripts that combine user input and sensing events and distribute output and across devices to create a range of rich cross-device behaviors with minimal development. The interaction system includes an integrated environment for developers to author and test cross-device behaviors. When a script is ready, the script can be deployed on a network of mobile and wearable computing devices using the interaction system'"'"'s runtime environment. An interaction system script can. An evaluation of the interaction system with twelve participants revealed that the interaction system significantly reduced developer effort for creating and iterating on cross-device behaviors. The interaction system allowed developers to focus on target interaction behaviors and high-level device capabilities, rather than low-level specifications.

Claims

1. A method, comprising:

receiving a script at an interaction system executing on a particular computing device, wherein the interaction system is configured to communicatively couple to one or more other computing devices; and

executing the script using the interaction system, wherein the script, when executed by the interaction system, is configured to cause the interaction system to perform interaction-system functions, and wherein the interaction-system functions comprise;

selecting, by the interaction system, a first set of two or more devices based on one or more first selection criteria provided in the script, wherein the first set of two or more devices is selected from the particular computing device and the one or more other computing devices, and wherein the one or more first selection criteria comprise a body-centric criterion indicating a location on a body,

determining, by the interaction system, that a first event has occurred, the first event related to a first designated device of the first set of two or more devices,

in response to the first event, determining, by the interaction system, a first operation to perform on the first designated device, and

instructing, by the interaction system, the first designated device to perform the first operation.

2. The method of claim 1, wherein the interaction-system functions further comprise:

determining, by the interaction system, that a second event has occurred, the second event related to a second designated device of the first set of two or more devices, wherein the first designated device differs from the second designated device;

in response to the second event, determining, by the interaction system, a second operation to perform on the second designated device; and

instructing, by the interaction system, the second designated device to perform the second operation.

3. The method of claim 1, wherein the interaction-system functions further comprise:

selecting, by the interaction system, a second set of two or more computing devices communicatively coupled to the particular computing device, wherein the first set differs from the second set;

determining, by the interaction system, that a third event has occurred, the third event related to a third designated device of the second set of two or more devices;

in response to the third event, determining, by the interaction system, a third operation to perform on the third designated device; and

instructing, by the interaction system, the third designated device to perform the third operation.

4. The method of claim 1, wherein at least one device of the first set of two or more devices is a wearable device.

5. The method of claim 4, wherein the wearable device is a device selected from the group of devices consisting of a smartphone, a smartwatch, and a head-mountable device (HMD).

6. The method of claim 1, wherein the first event indicates an input has been received from a first device of the first set of two or more devices, wherein the input relates to selected media to be displayed using the interaction system on a second device of the first set of two or more devices based on the input, wherein the first device and second device differ, and wherein the first operation comprises displaying the selected media on the second device.

7. The method of claim 1, wherein a particular device in the first set of two or more devices comprises one or more I/O device components, and wherein the first set of selection criteria comprise a selector keyword referring to an I/O device component of the one or more I/O device components.

8. The method of claim 7, wherein the one or more I/O device components include an I/O device component selected from the group of I/O device components consisting of:

a touchable I/O device component related to a touch input for a device,

a showable I/O device component related to visible output of a device,

a rotatable I/O device component related to a rotation of a device, and

a shakeable I/O device component related to an acceleration of a device.

9. The method of claim 1, wherein a pre-configured device is configured to store device-component information related to one or more I/O device components of the pre-configured device, and wherein the interaction-system functions further comprise:

receiving, at the interaction system, at least part of the device-component information about the pre-configured device; and

adding, by the interaction system, the pre-configured device to the first set of two or more devices using the device-component information about the pre-configured device.

10. The method of claim 1, wherein receiving the script comprises:

generating the script using a development user interface associated with the interaction system, and

testing the script, by the development user interface, on one or more emulated devices configured to represent one or more corresponding devices in the first set of two or more devices;

and wherein executing the script comprises using the interaction system for;

executing the script after testing the script on the one or more emulated devices.

11. The method of claim 1, wherein selecting the first set of two or more devices based on the one or more first selection criteria comprises generating, by the interaction system, a representation of a combination of devices in the first set of two or more devices configured to operate as a single conceptual device.

12. The method of claim 1, wherein selecting the first set of two or more devices based on the one or more first selection criteria comprises generating, by the interaction system, a representation of all devices in the first set of two or more devices configured to treat all devices equally.

13. A computing device, comprising:

a processor; and

a non-transitory computer readable medium configured to store executable instructions for at least an interaction system, wherein the interaction system is configured to communicatively couple to one or more other computing devices, and wherein the executable instructions, when executed by the processor, cause the interaction system of the computing device at least to perform functions comprising;

receive a script at the interaction system, and

execute the script using the interaction system, wherein the script is configured to cause the interaction system to perform interaction-system functions, and wherein the interaction-system functions comprise;

selecting, by the interaction system, a first set of two or more devices based on one or more first selection criteria provided in the script, wherein the first set of two or more devices is selected from the computing device and the one or more other computing devices, and wherein the one or more first selection criteria comprise a body-centric criterion indicating a location on a body,

determining, by the interaction system, that a first event has occurred, the first event related to a first designated device of the first set of two or more devices,

in response to the first event, determining, by the interaction system, a first operation to perform on the first designated device, and

instructing, by the interaction system, the first designated device to perform the first operation.

14. The computing device of claim 13, wherein the interaction-system functions further comprise:

determining, by the interaction system, that a second event has occurred, the second event related to a second designated device of the first set of two or more devices, wherein the first designated device differs from the second designated device;

in response to the second event, determining, by the interaction system, a second operation to perform on the second designated device; and

instructing, by the interaction system, the second designated device to perform the second operation.

15. The computing device of claim 13, wherein the interaction-system functions further comprise:

selecting, by the interaction system, a second set of two or more computing devices communicatively coupled to the computing device, wherein the first set differs from the second set;

determining, by the interaction system, that a third event has occurred, the third event related to a third designated device of the second set of two or more devices;

in response to the third event, determining, by the interaction system, a third operation to perform on the third designated device; and

instructing, by the interaction system, the third designated device to perform the third operation.

16. The computing device of claim 13, wherein at least one device of the first set of two or more devices is a wearable device.

17. The computing device of claim 16, wherein the wearable device is a device selected from the group of devices consisting of a smartphone, a smartwatch, and a head-mountable device (HMD).

18. The computing device of claim 13, wherein the first event indicates an input has been received from a first device of the first set of two or more devices, wherein the input relates to selected media to be displayed using the interaction system on a second device of the first set of two or more devices based on the input, wherein the first device and second device differ, and wherein the first operation comprises displaying the selected media on the second device.

19. The computing device of claim 13, wherein a particular device in the first set of two or more devices comprises one or more I/O device components, and wherein the first set of selection criteria comprise a selector keyword referring to an I/O device component of the one or more I/O device components.

20. The computing device of claim 13, wherein the one or more I/O device components include an I/O device component selected from the group of I/O device components consisting of:

a touchable I/O device component related to a touch input for a device,

a showable I/O device component related to visible output of a device,

a rotatable I/O device component related to a rotation of a device, and

a shakeable I/O device component related to an acceleration of a device.

21. The computing device of claim 20, wherein a pre-configured device is configured to store device-component information related to one or more I/O device components of the pre-configured device, and wherein the interaction-system functions further comprise:

receiving, at the interaction system, at least part of the device-component information about the pre-configured device; and

adding, by the interaction system, the pre-configured device to the first set of two or more devices using the device-component information about the pre-configured device.

22. The computing device of claim 13, wherein receiving the script comprises:

generating the script using a development user interface associated with the interaction system, and

testing the script, by the development user interface, on one or more emulated devices configured to represent one or more corresponding devices in the first set of two or more devices;

and wherein executing the script comprises using the interaction system for;

executing the script after testing the script on the one or more emulated devices.

23. A computer readable medium configured to store at least executable instructions, wherein the executable instructions, when executed by a processor of a computing device, cause the computing device to perform functions comprising:

receiving a script at an interaction system executing on the computing device, wherein the interaction system is configured to communicatively couple to one or more other computing devices; and

executing the script using the interaction system, wherein the script is configured to cause the interaction system to perform interaction-system functions, wherein the interaction-system functions comprise;

selecting, by the interaction system, a first set of two or more devices based on one or more first selection criteria provided in the script, wherein the first set of two or more devices is selected from the computing device and the one or more other computing devices, and wherein the one or more first selection criteria comprise a body-centric criterion indicating a location on a body,

determining, by the interaction system, that a first event has occurred, the first event related to a first designated device of the first set of two or more devices,

in response to the first event, determining, by the interaction system, a first operation to perform on the first designated device, and

instructing, by the interaction system, the first designated device to perform the first operation.

24. The computer readable medium of claim 23, wherein the interaction-system functions further comprise:

determining, by the interaction system, that a second event has occurred, the second event related to a second designated device of the first set of two or more devices, wherein the first designated device differs from the second designated device;

in response to the second event, determining, by the interaction system, a second operation to perform on the second designated device; and

instructing, by the interaction system, the second designated device to perform the second operation.

25. The computer readable medium of claim 23, wherein the interaction-system functions further comprise:

determining, by the interaction system, that a second event has occurred, the second event related to a second designated device of the first set of two or more devices, wherein the first designated device differs from the second designated device;

in response to the second event, determining, by the interaction system, a second operation to perform on the second designated device; and

instructing, by the interaction system, the second designated device to perform the second operation.