External visual interactions for speech-based devices

• US 11,327,711 B2
• Filed: 12/05/2014
• Issued: 05/10/2022
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, Weidong, Caldwell, Gary, Bhargava, Ruchi
• Primary Examiner(s): Paula, Cesar B
• Assistant Examiner(s): Barnes, Jr., Carl E
• Application Number:
  US14/562,491
• Publication Number:
  US 20160162259A1
• Time in Patent Office: 2,713 Days
• Field of Search: 715243-246, 715251-252, 715234
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58

Abstract

Examples are disclosed herein that are related to providing extended functionalities on-demand to an audio-based wearable device. One example provides a wearable computing device including an acoustic receiver configured to receive speech inputs, a speaker configured to present audio outputs, a communications subsystem configured to connect to an external device, a logic subsystem configured to execute instructions, and a storage subsystem having instructions executable by the logic subsystem to execute a program, connect to the external device via a wireless communications protocol, conduct an audio-based interaction of the program via the speech inputs received at the acoustic receiver and the audio outputs provided by the speaker, upon reaching a screen-based interaction of the program, notify a user via the speaker to interact with the external device, and provide image data to the external device for presentation via a screen of the external device.

Claims

1. A wearable computing device in the form of an earpiece and not having a display screen, the earpiece comprising:

an acoustic receiver configured to receive speech inputs;

a speaker configured to present audio outputs;

a communications subsystem configured to connect to an external extender device comprising a display;

a logic subsystem residing on the wearable computing device, the logic subsystem configured to execute instructions; and

a storage subsystem residing on the wearable computing device, the storage subsystem comprising instructions executable by the logic subsystem to

execute a program on the wearable computing device, the program being executable to receive the speech inputs via the acoustic receiver of the earpiece and to provide outputs via the speaker of the earpiece,

connect to the external extender device via a wireless communications protocol,

conduct an audio-based interaction of the program being executed on the wearable computing device via the speech inputs received at the acoustic receiver of the earpiece and the audio outputs provided by the speaker of the earpiece,

while conducting the audio-based interaction, upon reaching a screen-based interaction of the program being executed on the wearable computing device, output via the speaker of the earpiece a notification notifying a user of the screen-based interaction, and notify the user via audio output to interact with the external extender device during the screen-based interaction, the screen-based interaction comprising display of an image received from a remote computing device; and

via the wireless communications protocol, send from the wearable computing device to the external extender device one or more of image data and an instruction to obtain image data from a remote service to the external extender device for presentation via the display of the external extender device.

2. The wearable computing device of claim 1, wherein the wireless communications protocol is a first wireless communications protocol, and wherein the instructions are executable by the logic subsystem to connect to the external extender device also via a second wireless communications protocol.

3. The wearable computing device of claim 2, wherein the instructions are executable by the logic subsystem to monitor one or more performance characteristics for each of the first wireless communications protocol and the second wireless communications protocol, and to communicate with the external extender device via one of the first wireless communications protocol and the second wireless communications protocol based upon the one or more performance characteristics.

4. The wearable computing device of claim 1, wherein the instructions are executable by the logic subsystem to connect to the external extender device by detecting a beacon signal emitted by the external extender device.

5. The wearable computing device of claim 1, wherein the instructions are executable by the logic subsystem to connect to one or more additional external devices.

6. The wearable computing device of claim 1, wherein the instructions are executable by the logic subsystem to receive from the external extender device information regarding one or more of an orientation, a frame rate, and a resolution of the display of the external extender device.

7. The wearable computing device of claim 6, wherein the instructions are executable by the logic subsystem to receive the image data from the remote computing device and send from the earpiece to the external extender device the image data received from the remote computing device.

8. The wearable computing device of claim 1, wherein the instructions are executable to receive user inputs for the program made via the external extender device, and wherein the user inputs comprise position signals corresponding to touch inputs on the external extender device.

9. A wearable computing device in the form of an earpiece and not having a display screen, the earpiece configured to utilize acoustic interactions as a primary user interface experience and comprising:

an acoustic receiver configured to receive speech inputs;

a speaker configured to present audio outputs;

a communications subsystem configured to connect to an external extender device and a remote service, the external extender device comprising a display;

a logic subsystem residing on the wearable computing device, the logic subsystem configured to execute instructions; and

a storage subsystem residing on the wearable computing device, the storage subsystem comprising instructions executable by the logic subsystem to

connect to the remote service via a computer network,

connect to the external extender device,

execute a program locally on the earpiece as the primary user interface experience, the program being executable to receive the speech inputs via the acoustic receiver of the earpiece and to provide outputs via the speaker of the earpiece,

conduct an audio-based interaction of the program being executed on the wearable computing device via the speech inputs received at the acoustic receiver of the earpiece and the audio outputs provided by the speaker of the earpiece,

while conducting the audio-based interaction, upon reaching a screen-based interaction of the program being executed on the wearable computing device, output via the speaker of the earpiece a notification notifying a user of the screen-based interaction, and notify the user via audio output to interact with the external extender device during the screen-based interaction, the screen-based interaction comprising display of an image received from a remote computing device;

send an instruction from the earpiece to the external extender device to obtain image data from the remote service for the screen-based interaction, and

redirect user input information from the external extender device to the earpiece during the screen-based interaction.

10. The wearable computing device of claim 9, wherein the instructions are executable by the logic subsystem to connect to the external extender device via a first wireless communications protocol and a second wireless communications protocol, to monitor one or more performance characteristics for each of the first wireless communications protocol and the second wireless communications protocol, and to communicate with the external extender device via one of the first wireless communications protocol and the second wireless communications protocol based upon the one or more performance characteristics.

11. The wearable computing device of claim 9, wherein the instructions are executable by the logic subsystem to receive the user input information from the external extender device via the remote service.

12. The wearable computing device of claim 9, wherein the instructions are executable by the logic subsystem to receive the user input information directly from the external extender device.

13. The wearable computing device of claim 12, wherein the instructions are executable by the logic subsystem to forward the user input information received directly from the external extender device to the remote service.

14. The wearable computing device of claim 9, wherein the instructions are executable by the logic subsystem to send image data from the earpiece to the external extender device.

15. The wearable computing device of claim 9, wherein the user input information comprises position signals corresponding to touch inputs on the external extender device.

16. On a wearable computing device in the form of an earpiece and not having a display screen, the earpiece configured to utilize acoustic interactions as a primary user interface experience and comprising an acoustic receiver and a speaker, a method comprising:

executing a program on the earpiece as the primary user interface experience, the program being executable to receive speech inputs via the acoustic receiver of the earpiece and to provide outputs via the speaker of the earpiece;

connecting to an external extender device comprising a display via a wireless communications protocol;

conducting an audio-based interaction of the program being executed on the earpiece via the speech inputs received at the acoustic receiver of the earpiece and the audio outputs provided by the speaker of the earpiece;

while conducting the audio-based interaction, upon reaching a screen-based interaction of the program being executed on the earpiece, outputting via the speaker of the earpiece a notification notifying a user of the screen-based interaction and to interact with the external extender device during the screen-based interaction, the screen-based interaction comprising display of an image received from a remote computing device;

receiving from the external extender device orientation information regarding an orientation of the display of the external extender device;

via the wireless communications protocol, providing image data to the external extender device based upon the orientation information for presentation via the display of the external extender device; and

receiving user input information from the external extender device during the screen-based interaction.

17. The method of claim 16, wherein the wireless communications protocol is a first wireless communications protocol, and further comprising connecting to the external extender device also via a second wireless communications protocol.

18. The method of claim 17, further comprising monitoring one or more performance characteristics for each of the first wireless communications protocol and the second wireless communications protocol, and communicating with the external extender device via one of the first wireless communications protocol and the second wireless communications protocol based upon the one or more performance characteristics.

19. The method of claim 16, wherein the user input information comprises position signals corresponding to touch inputs on the external extender device.

20. The method of claim 16, further comprising connecting a remote service via a computer network, wherein receiving the user input information from the external extender device comprises receiving the user input information via the remote service.