HANDS-FREE NAVIGATION OF TOUCH-BASED OPERATING SYSTEMS

• US 20180181370A1
• Filed: 12/23/2016
• Published: 06/28/2018
• Est. Priority Date: 12/23/2016
• Status: Active Grant

• Current Assignee: RealWear, Inc.
• Original Assignee: RealWear, Inc.
• Inventors: PARKINSON, CHRISTOPHER IAIN
• Granted Patent: US 10,620,910 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Aspects of the present invention allow hands-free navigation of touch-based operating systems. The system may automatically interface with a touch-based operating system and generate hands-free commands that are associated with touch-based commands. Embodiments may utilize motion and/or audio inputs to facilitate hands-free interaction with the touch-based operating system.

Claims

1. A computer-implemented method for hands-free navigation of a head-mounted computing device, the method comprising:

initializing a hands-free interaction mode on the head-mounted computing device, wherein the hands-free interaction mode enables interaction with a user interface of a touch-based operating system;

detecting an initial position of the head-mounted computing device, the initial position including a position of the head-mounted computing device relative to one or more axes;

displaying a first instance of a user interface of the touch-based operating system on a display of the head-mounted computing device;

detecting a first displacement of the head-mounted computing device, the first displacement being above a first threshold displacement, wherein the first threshold displacement is in displacement relative to the initial position;

determining a first instruction based on the first displacement, wherein the first instruction corresponds to one or more touch-based inputs; and

communicating the first instruction to the touch-based operating system.

2. The computer-implemented method of claim 1, further comprising receiving a first audible input associated with the user interface of the touch-based operating system and determining a second instruction based on the first audible input.

3. The computer-implemented method of claim 1, wherein the user interface is associated with an electronic document in the touch-based operating system.

4. The computer-implemented method of claim 2, wherein the first instruction comprises a scrolling instruction having a fixed number of scroll units.

5. The computer-implemented method of claim 1, wherein the first threshold displacement corresponds to a minimum angular displacement of the head-mounted computing device relative to the one or more axes.

6. The computer-implemented method of claim 1, wherein the first displacement is detected by one or more sensors of the head-mounted computing device.

7. The computer-implemented method of claim 1, wherein detecting the first displacement comprises determining a duration of the first displacement.

8. The computer-implemented method of claim 1, further comprising displaying a second instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

9. The computer-implemented method of claim 8, further comprising:

detecting a second displacement of the head-mounted computing device, the second displacement being above a second threshold displacement;

determining a second instruction based on the second displacement, wherein the second instruction corresponds to the one or more touch-based inputs; and

displaying a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

10. A non-transitory computer storage medium storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

initializing a hands-free interaction mode on the head-mounted computing device, wherein the hands-free interaction mode enables interaction with a user interface of a touch-based operating system;

identifying at least one touch-based control dialogue of the user interface of the touch-based operating system;

displaying a first instance of the user interface of the touch-based operating system on a display of the head-mounted computing device;

receiving a first audible input associated with the at least one touch-based control dialogue; and

displaying a second instance of the user interface of the touch-based operating system on the head-mounted computing device in response to the first audible input.

11. The computer storage medium of claim 10, further comprising displaying at least one graphical overlay control over the first instance of the user interface on the display of the head-mounted computing device.

12. The computer storage medium of claim 11, wherein the at least one graphical overlay control includes a visual indicator that represents an audible input corresponding to the at least one touch-based control dialogue.

13. The computer storage medium of claim 10, further comprising selecting the at least one graphical overlay control based on identifying a graphical overlay control of a plurality of predetermined graphical overlay controls associated with the at least one touch-based control dialogue.

14. The computer storage medium of claim 10, further comprising receiving a second audible input, the second audible input corresponding to an instruction to suspend the hands-free interaction mode.

15. The computer storage medium of claim 10, further comprising performing speech recognition to recognize text from the first audible input.

16. The computer storage medium of claim 10, wherein the user interface is associated with a mobile application executing in the touch-based operating system.

17. A system for hands-free navigation of a head-mounted computing device, the system comprising:

one or more processors; and

one or more computer storage media storing computer-useable instructions that, when executed by the one or more processors, implement a method comprising;

initializing a hands-free interaction mode on the head-mounted computing device, wherein the hands-free interaction mode enables interaction with a user interface of a touch-based operating system;

displaying a first instance of a user interface of the touch-based operating system on a display of the head-mounted computing device;

detecting a first hands-free input comprising one or more of;

a first displacement of the head-mounted computing device, the first displacement being above a threshold displacement; and

a first audible input associated with a touch-based control dialogue included in the first instance of the user interface;

determining a first instruction based on the hands-free input, wherein the first instruction corresponds to one or more touch-based inputs; and

displaying a second instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

18. The system of claim 17, wherein the first hands-free input comprises the first displacement and the first audible input.

19. The system of claim 17, wherein the implemented method further comprises:

detecting a second hands-free input;

determining a second instruction based on the second hands-free input, wherein the second instruction corresponds to the one or more touch-based inputs; and

displaying a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

20. The system of claim 17, wherein the implemented method further comprises detecting a hands-free initialization input and generating an instruction to initialize the hands-free interaction mode on the head-mounted computing device.