SYSTEMS, DEVICES, AND METHODS FOR INTERACTING WITH CONTENT DISPLAYED ON HEAD-MOUNTED DISPLAYS

• US 20170097753A1
• Filed: 09/30/2016
• Published: 04/06/2017
• Est. Priority Date: 10/01/2015
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Thalmic Labs Inc.
• Inventors: Bailey, Matthew, Alexander, Stefan
• Application Number:
  US15/282,535
• Publication Number:
  US 20170097753A1
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser

Abstract

Systems, devices, and methods that enable sophisticated and inconspicuous interactions with content displayed on a head-mounted display are described. A head-mounted display includes an eye-tracker and the user also carries/wears a wireless portable interface device elsewhere on their body, such as a ring. The wireless nature of the portable interface device enables a small and unobtrusive form factor. The portable interface device includes an actuator that, when activated by the user, causes the portable interface device to wirelessly transmit a signal (e.g., a radio frequency signal or a sonic signal). A selection operation performed by the user is defined as the user activating the actuator of the portable interface device while the user is substantially concurrently gazing at a displayed object (as detected by the eye-tracker). In response to the selection operation, the head-mounted display displays a visual effect to the user.

Claims

1. A system that enables interaction with content displayed on a head-mounted display, the system comprising:

a head-mounted display including;

at least one display positioned in a field of view of at least one eye of a user when the head-mounted display is worn on a head of the user;

a processor communicatively coupled to the at least one display;

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable instructions and/or data that, when executed by the processor, cause the at least one display to display at least one object that is responsive to a selection operation performed by the user; and

a wireless receiver to wirelessly receive signals, the wireless receiver communicatively coupled to the processor;

an eye-tracker communicatively coupled to the processor, the eye-tracker to detect that the user is gazing at the at least one object displayed by the at least one display; and

a wireless portable interface device with a form factor to be carried by or on the user, the portable interface device physically separate from the head-mounted display, wherein the portable interface device includes at least one actuator that, when activated by the user, causes the portable interface device to wirelessly transmit a signal.

2. The system of claim 1 wherein:

the selection operation performed by the user comprises a substantially concurrent combination of gazing at the least one object displayed by the at least one display and activating the at least actuator of the portable interface device; and

in response to the selection operation performed by the user, a visual effect is displayed on the at least one display of the head-mounted display.

3. The system of claim 2 wherein the non-transitory processor-readable storage medium further stores processor-executable instructions and/or data that, when executed by the processor in response to the wireless receiver wirelessly receiving the signal from the portable interface device, cause the processor to:

request current gaze direction data from the eye-tracker;

identify a particular object at which the user is gazing based on the current gaze direction data received from the eye-tracker, the particular object identified among the at least one object displayed by the at least one display; and

cause the at least one display to display the visual effect on the particular object.

4. The system of claim 1 wherein the portable interface device is batteryless and includes a piezoelectric element communicatively coupled to the actuator and an antenna communicatively coupled to the piezoelectric element, and wherein the wireless receiver of the head-mounted display includes a radio frequency receiver.

5. The system of claim 4 wherein:

when activated by the user, the actuator mechanically actuates the piezoelectric element,

in response to the mechanical actuation, the piezoelectric element generates an electric signal, and

in response to the electric signal generated by the piezoelectric element, the antenna transmits a radio frequency signal.

6. The system of claim 1 wherein the portable interface device is batteryless and includes a mechanical resonator physically coupled to the actuator, and wherein the wireless receiver of the head-mounted display includes at least one of:

a microphone and/or a piezoelectric element tuned to be responsive to a sonic signal.

7. The system of claim 6 wherein:

when activated by the user, the actuator mechanically actuates the mechanical resonator, and

in response to the mechanical actuation, the mechanical resonator generates the sonic signal.

8. The system of claim 1 wherein the portable interface device includes an on-board power source and a radio frequency transmitter to transmit radio signals having a frequency in a frequency range of 10 MHz to 10 GHz, and wherein the wireless receiver of the head-mounted display includes a radio frequency receiver tuned to receive the radio signals.

9. The system of claim 7 wherein the sonic signal includes an ultrasonic signal.

10. The system of claim 1 wherein the object includes at least one object selected from a group consisting of:

a menu item, a graphical button, a keyboard key, a notification, one of multiple objects displayed by the at least one display of the head-mounted display, a file, a folder, and an alphanumeric character.

11. The system of claim 1 wherein the portable interface device is a wearable device selected from a group consisting of:

a ring, a wristband, and an armband.

12. The system of claim 1 wherein the actuator of the portable interface device includes a button.

13. The system of claim 1 wherein the eye-tracker is carried by and physically coupled to the head-mounted display.

14. A method of operating a system, wherein the system comprises a head-mounted display, an eye-tracker, and a wireless portable interface device, the portable interface device physically separate from the head-mounted display, the method comprising:

displaying an object within a field of view of at least one eye of a user by at least one display of the head-mounted display;

receiving a selection operation from the user by the system, wherein receiving a selection operation from the user by the system includes;

detecting, by the eye tracker, that the user is gazing at the object;

while detecting, by the eye tracker, that the user is gazing at the object, receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device;

in response to receiving, by the portable interface device, the activation from the user of the actuator of the portable interface device, wirelessly transmitting a signal by the portable interface device; and

wirelessly receiving the signal by a wireless receiver of the head-mounted display; and

in response to receiving the selection operation from the user by the system, displaying a visual effect by the at least one display of the head-mounted display.

15. The method of claim 14 wherein the head-mounted display includes a processor and a non-transitory processor-readable storage medium communicatively coupled to the processor, and wherein the non-transitory processor-readable storage medium stores processor-executable instructions and/or data and the method comprises executing, by the processor, the processor-executable instructions and/or data stored in the non-transitory processor-readable storage medium to cause:

the at least one display of the head-mounted display to display the object within the field of view of at least one eye of the user; and

the at least one display of the head-mounted display to display the visual effect in response to the system receiving the selection operation from the user.

16. The method of claim 14 wherein:

the portable interface device is batteryless and includes a piezoelectric element communicatively coupled to the actuator and an antenna communicatively coupled to the piezoelectric element,

the wireless receiver of the head-mounted display includes a radio receiver,

receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device includes receiving, by the portable interface device, a mechanical actuation of the piezoelectric element from the user,

in response to the mechanical actuation, the piezoelectric element generates an electric signal,

wirelessly transmitting a signal by the portable interface device includes wirelessly transmitting a radio signal by the antenna of the portable interface device in response to the electric signal generated by the piezoelectric element, and

wirelessly receiving the signal by a wireless receiver of the head-mounted display includes wirelessly receiving the radio signal by the radio receiver of the head-mounted display.

17. The method of claim 14 wherein:

the portable interface device is batteryless and includes a mechanical resonator physically coupled to the actuator,

receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device includes receiving, by the portable interface device, a mechanical actuation of the mechanical resonator from the user,

in response to the mechanical actuation, the mechanical resonator generates a sonic signal,

wirelessly transmitting a signal by the portable interface device includes wirelessly transmitting the sonic signal by mechanical resonator of the portable interface device, and

wirelessly receiving the signal by a wireless receiver of the head-mounted display includes wirelessly receiving the sonic signal by the wireless receiver of the head-mounted display, and wherein the wireless receiver of the head-mounted display includes at least one of;

a microphone and/or a piezoelectric element tuned to be responsive to the sonic signal.

18. The method of claim 17 wherein the sonic signal generated by the mechanical resonator of the portable interface device includes an ultrasonic signal.

19. The method of claim 14 wherein:

the portable interface device includes an on-board power source and a radio frequency transmitter;

wirelessly transmitting a signal by the portable interface device includes wirelessly transmitting a radio signal by the radio frequency transmitter of the portable interface device, the radio signal having a frequency in a range of 10 MHz to 10 GHz, and

wirelessly receiving the signal by a wireless receiver of the head-mounted display includes wirelessly receiving the radio signal by the wireless receiver of the head-mounted display, and wherein the wireless receiver of the head-mounted display includes a radio frequency receiver.

20. The method of claim 14 wherein the object includes at least one object selected from a group consisting of:

a menu item, a graphical button, a keyboard key, a notification, one of multiple objects displayed by the at least one display of the head-mounted display, a file, a folder, and an alphanumeric character.

21. The method of claim 14 wherein the portable interface device is a wearable device selected from a group consisting of:

a ring, a wristband, and an armband.

22. A completely wearable human-electronics interface comprising:

a wearable heads-up display including a wireless receiver;

an eye-tracker carried by the wearable heads-up display; and

a wearable actuator including a wireless transmitter to transmit wireless signals to the wireless receiver of the wearable heads-up display.

23. The completely wearable human-electronics interface of claim 22 wherein the wearable heads-up display further includes a processor that is communicatively coupled to both the eye-tracker and the wireless receiver, the processor to effect interactions with content displayed by the wearable heads-up display in response to concurrent inputs from both the eye-tracker and the wearable actuator.