Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display

• US 10,528,135 B2
• Filed: 01/14/2014
• Issued: 01/07/2020
• Est. Priority Date: 01/14/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Bailey, Matthew, Grant, Aaron, Lake, Stephen
• Primary Examiner(s): Hicks, Charles V
• Application Number:
  US14/155,107
• Publication Number:
  US 20140198035A1
• Time in Patent Office: 2,184 Days
• Field of Search: 345156
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems, devices and methods that enable a user to access and interact with content displayed on a portable electronic display in an inconspicuous, hands-free manner are described. There is disclosed a completely wearable system comprising a wearable muscle interface device and a wearable head-mounted display, as well as methods for using the wearable system to effect interactions between the user and content displayed on the wearable head-mounted display. The wearable muscle interface device includes muscle activity sensors worn on an arm of the user to detect muscle activity generated when the user performs a physical gesture. The wearable system is adapted to recognize a plurality of gestures made by the user and, in response to each recognized gesture, to effect one or more interaction(s) with content displayed on the wearable head-mounted display.

Claims

1. A wearable muscle interface device for controlling navigable content displayed on a wearable head-mounted display, the wearable muscle interface device comprising:

a plurality of electromyography sensors to be worn on an arm of a user, the plurality of electromyography sensors for detecting electrical signals of muscle activity corresponding to a gesture performed by the user;

at least one accelerometer for detecting motion of the arm of the user corresponding to the gesture performed by the user;

a first processor, communicatively coupled to the plurality of electromyography sensors and the at least one accelerometer, for;

generating, based on at least the electrical signals detected by the plurality of electromyography sensors and the motion of the arm of the user detected by the at least one accelerometer,

a first interpretation of the gesture performed by the user, and

a second interpretation of the gesture performed by the user, and

generating at least one control signal to request input from the user to select one of the first interpretation of the gesture or the second interpretation of the gesture,

wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the first processor from a set of at least three defined gestures; and

a transmitter communicatively coupled to the first processor, wherein the transmitter transmits the at least one control signal from the wearable muscle interface device directly to a receiver on the wearable head-mounted display,

wherein the at least one control signal causes a second processor of the wearable head-mounted display to execute data and/or instructions stored in a non-transitory computer-readable storage medium of the wearable head-mounted display to effect at least one control interaction including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

2. The wearable muscle interface device of claim 1, further comprising a haptic feedback module for providing haptic feedback to the user, the haptic feedback module including a vibratory motor.

3. The wearable muscle interface device of claim 1 wherein the transmitter includes a wireless transmitter.

4. A wearable system for providing hands-free access to and control of a portable electronic display, the wearable system comprising:

i) a wearable muscle interface device comprising;

a plurality of electromyography sensors to be worn on an arm of a user, the plurality of electromyography sensors for detecting electrical signals of muscle activity corresponding to a gesture performed by the user;

at least one accelerometer for detecting motion of the arm of the user corresponding to the gesture performed by the user;

a first processor communicatively coupled to the plurality of electromyography sensors and the at least one accelerometer, for;

generating, based on at least the electrical signals detected by the plurality of electromyography sensors and the motion of the arm of the user detected by the at least one accelerometer,

a first interpretation of the gesture performed by the user, and

a second interpretation of the gesture performed by the user, and

generating at least one control signal to request input from the user to select one of the first interpretation of the gesture or the second interpretation of the gesture,

wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the first processor from a set of at least three defined gestures; and

a transmitter communicatively coupled to the first processor, wherein the transmitter transmits the at least one control signal from the wearable muscle interface device; and

ii) a wearable head-mounted display comprising;

at least one display screen to be worn on a head of the user, the at least one display screen arranged to be positioned in front of at least one eye of the user when worn on the head of the user;

a receiver communicatively coupled to the at least one display screen, wherein the receiver directly receives the at least one control signal transmitted from the transmitter of the wearable muscle interface device;

a second processor communicatively coupled to the receiver and to the at least one display screen;

a non-transitory computer-readable storage medium communicatively coupled to the processor; and

data and/or instructions stored in the non-transitory computer-readable storage medium that, when executed by the second processor in response to the at least one control signal, cause the second processor to effect at least one control interaction including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

5. The wearable system of claim 4 wherein the transmitter of the wearable muscle interface device includes a wireless transmitter and the receiver of the wearable head-mounted display includes a wireless receiver.

6. A method of using a wearable system to achieve hands-free access to and control of a portable electronic display, wherein the wearable system includes a wearable muscle interface device to be worn on an arm of the user and a wearable head-mounted display, the method comprising:

detecting electrical signals of muscle activity corresponding to a gesture performed by a user of the wearable system by at least one electromyography sensor of the wearable muscle interface device;

detecting motion of the arm of the user corresponding to the gesture performed by the user of the wearable system, by at least one accelerometer of the wearable muscle interface device;

generating, by a first processor of the wearable muscle interface device, based on at least the electrical signals detected by the at least one electromyography sensor of the wearable muscle interface device and the motion of the arm of the user detected by the at least one accelerometer of the wearable muscle interface device,

a first interpretation of the gesture performed by the user, and

a second interpretation of the gesture performed by the user,

wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the first processor from a set of at least three defined gestures;

generating, by the first processor, a control signal to request input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user;

transmitting the at least one control signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device;

receiving the at least one control signal directly from the wearable muscle interface device by a receiver of the wearable head-mounted display; and

processing the at least one control signal by a second processor of the wearable head-mounted display by;

executing, by the second processor of the wearable head-mounted display, data and/or instructions stored in a non-transitory computer-readable storage medium of the wearable head-mounted display to effect at least one control interaction including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

7. The method of claim 6 wherein:

transmitting the at least one control signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device includes wirelessly transmitting the at least one control signal from the wearable muscle interface device by a wireless transmitter of the wearable muscle interface device; and

receiving the at least one control signal directly from the wearable muscle interface device by a receiver of the wearable head-mounted display includes wirelessly receiving the at least one control signal directly from the wearable muscle interface device by a wireless receiver of the wearable head-mounted display.