Hands-free navigation of touch-based operating systems

• US 10,620,910 B2
• Filed: 12/23/2016
• Issued: 04/14/2020
• Est. Priority Date: 12/23/2016
• Status: Active Grant

• Current Assignee: RealWear, Inc.
• Original Assignee: RealWear, Inc.
• Inventors: Parkinson, Christopher Iain
• Primary Examiner(s): Theriault, Steven B
• Application Number:
  US15/390,191
• Publication Number:
  US 20180181370A1
• Time in Patent Office: 1,208 Days
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Aspects of the present invention allow hands-free navigation of touch-based operating systems. The system may automatically interface with a touch-based operating system and generate hands-free commands that are associated with touch-based commands. Embodiments may utilize motion and/or audio inputs to facilitate hands-free interaction with the touch-based operating system.

Claims

1. A computer-implemented method for hands-free navigation of a head-mounted computing device, the method comprising:

initializing, by the head-mounted computing device (HMD), a hands-free interaction mode, wherein the hands-free interaction mode enables hands-free interaction with a user interface of a touch-based operating system of the HMD;

based on the hands-free interaction mode being initialized, detecting, by the HMD, an initial position of the head-mounted computing device based on sensor data received from a set of sensors coupled to the HMD;

based on the initial position being detected, displaying, by the HMD, a first instance of a user interface of the touch-based operating system on a display of the head-mounted computing device, the first instance of the user interface being displayed while the HMD is in the detected initial position;

determining, by the HMD, that a first displacement of the HMD is greater than a defined first threshold displacement relative to the detected initial position, wherein the first displacement is determined based on additional sensor data received from the set of sensors;

converting, by the HMD, the first displacement to a corresponding set of touch-based inputs for communication to the touch-based operating system based on the determination that the first displacement is greater than the defined first threshold displacement.

2. The computer-implemented method of claim 1, further comprising receiving a first audible input associated with the user interface of the touch-based operating system and converting the first audible input to another corresponding set of touch-based inputs.

3. The computer-implemented method of claim 1, wherein the user interface is associated with an electronic document in the touch-based operating system.

4. The computer-implemented method of claim 2, wherein the set of touch-based inputs comprises a scrolling instruction having a fixed number of scroll units.

5. The computer-implemented method of claim 1, wherein the defined first threshold displacement corresponds to a minimum angular displacement of the head-mounted computing device relative to the initial position.

6. The computer-implemented method of claim 1, wherein the set of sensors comprise at least one of a camera, accelerometer, gyroscope, magnetometer first displacement is detected by one or more sensors of the head-mounted computing device.

7. The computer-implemented method of claim 1, wherein detecting the first displacement comprises determining a duration of the first displacement.

8. The computer-implemented method of claim 1, further comprising displaying, by the HMD, a second instance of the user interface of the touch-based operating system on the display of the head-mounted computing device in response to the converted first displacement.

9. The computer-implemented method of claim 8, further comprising:

detecting, by the HMD, a second displacement of the head-mounted computing device, the second displacement being above a defined second threshold displacement;

converting, by the HMD, a second instruction based on the detected second displacement, wherein the second instruction corresponds to the one or more touch-based inputs; and

displaying, by the HMD, a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

10. A system for hands-free navigation of a head-mounted computing device (HMD), the system comprising:

one or more processors; and

one or more computer storage media storing computer-useable instructions that, when executed by the one or more processors, implement a method comprising;

initializing a hands-free interaction mode, wherein the hands-free interaction mode enables hand-free interaction with a user interface of a touch-based operating system of the HMD;

displaying a first instance of a user interface of the touch-based operating system on a display of the HMD;

detecting, while the first instance of the user interface is being displayed, a first hands-free input comprising one or more of;

a first displacement of the head-mounted computing device based on sensor data received from a set of sensors coupled to the HMD, the first displacement being above a defined threshold displacement, or

a first audible input corresponding to a touch-based control dialogue included in the displayed first instance of the user interface;

converting the detected first hands-free input to a corresponding set of touch-based inputs for communication to the touch-based operating system; and

displaying a second instance of the user interface of the touch-based operating system on the display of the HMD.

11. The system of claim 10, wherein the first hands-free input comprises the first displacement and the first audible input.

12. The system of claim 10, wherein the implemented method further comprises:

detecting a second hands-free input;

determining a second instruction based on the second hands-free input, wherein the second instruction corresponds to the one or more touch-based inputs; and

displaying a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device.

13. The system of claim 10, wherein the implemented method further comprises detecting a hands-free initialization input and generating an instruction to initialize the hands-free interaction mode on the head-mounted computing device.