Systems, devices, and methods for wearable electronic devices as state machines

• US 10,199,008 B2
• Filed: 03/26/2015
• Issued: 02/05/2019
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: North Inc.
• Inventors: Aleem, Idris S., Ataee, Pedram, Lake, Stephen
• Primary Examiner(s): Lee, Benjamin C
• Assistant Examiner(s): Liang, Dong Hui
• Application Number:
  US14/669,878
• Publication Number:
  US 20150277575A1
• Time in Patent Office: 1,412 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

Systems, devices, and methods that implement state machine models in wearable electronic devices are described. A wearable electronic device stores processor-executable gesture identification instructions that, when executed by an on-board processor, enable the wearable electronic device to identify one or more gesture(s) performed by a user. The wearable electronic device also stores processor-executable state determination instructions that, when executed by the processor, cause the wearable electronic device to enter into and transition between various operational states depending on signals detected by on-board sensors. The state machine models described herein enable the wearable electronic devices to identify and automatically recover from operational errors, malfunctions, or crashes with minimal intervention from the user.

Claims

1. A wearable electronic device operable to automatically identify and recover from operational errors, the wearable electronic device comprising:

at least one sensor responsive to at least one input effected by a user of the wearable electronic device, wherein in response to the at least one input the at least one sensor provides sensor signals;

a processor communicatively coupled to the at least one sensor; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable sensor signal processing instructions and processor-executable state determination instructions, wherein the state determination instructions include a variable that determines an operational state of the processor, and wherein, when the state determination instructions are executed by the processor, the state determination instructions cause the processor to;

automatically identify when the wearable electronic device encounters an operational error based on a determination that the sensor signal processing instructions are not calibrated; and

automatically recover from the operational error, wherein to automatically recover from the operational error the state determination instructions cause the processor to;

enter a standby state automatically in response to the determination that the sensor signal processing instructions are not calibrated, wherein in the standby state the processor recognizes a first indication from the user, wherein the first indication from the user is a rest gesture performed by the user, the rest gesture indicative that the user is ready to calibrate the wearable electronic device;

enter a calibration state automatically in response to a recognition of the first indication from the user while the processor is in the standby state, wherein in the calibration state the processor recognizes a second indication from the user and calibrates the sensor signal processing instructions in response to the second indication from the user; and

enter an active state automatically in response to calibrating the sensor signal processing instructions while the processor is in the calibration state, wherein in the active state the processor executes the calibrated sensor signal processing instructions, and wherein when executed by the processor the calibrated sensor signal processing instructions cause the processor to process at least one input effected by the user based at least in part on the calibration of the sensor signal processing instructions from the calibration state.

2. The wearable electronic device of claim 1 wherein the at least one sensor is responsive to at least one gesture performed by the user and provides sensor signals in response to the at least one gesture, and wherein the processor-executable sensor signal processing instructions include processor-executable gesture identification instructions that, when executed by the processor while the processor is in the active state, cause the processor to identify at least one gesture performed by the user based at least in part on the calibration of the gesture identification instructions from the calibration state.

3. The wearable electronic device of claim 2 wherein the first indication from the user includes a recognition by the processor that the user has donned the wearable electronic device, and wherein the non-transitory processor-readable storage medium further stores processor-executable instructions that, when executed by the processor, cause the processor to recognize when the user dons the wearable electronic device as the first indication from the user in the standby state.

4. The wearable electronic device of claim 2 wherein the non-transitory processor-readable storage medium further stores processor-executable instructions that, when executed by the processor, cause the processor to:

recognize at least one reference gesture performed by the user as the second indication from the user in the calibration state; and

calibrate the gesture identification instructions based on the at least one reference gesture.

5. The wearable electronic device of claim 2 wherein the at least one sensor includes at least one muscle activity sensor selected from the group consisting of:

an electromyography (“

EMG”

) sensor and a mechanomyography (“

MMG”

) sensor.

6. The wearable electronic device of claim 1 wherein the at least one sensor includes at least one inertial sensor selected from the group consisting of:

an inertial measurement unit, an accelerometer, and a gyroscope.

7. The wearable electronic device of claim 1, further comprising:

at least one light-emitting diode (“

LED”

) communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium further stores processor-executable instructions that, when executed by the processor, cause the wearable electronic device to activate at least one of;

a first color of the at least one LED in response to the processor entering the standby state;

a second color of the at least one LED in response to the processor entering the calibration state; and

/or

a third color of the at least one LED in response to the processor entering the active state.

8. The wearable electronic device of claim 1, further comprising:

a haptic feedback device communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium further stores processor-executable instructions that, when executed by the processor, cause the processor to activate the haptic feedback module in response to at least one of;

entering the standby state;

entering the calibration state; and

/or

entering the active state.

9. The wearable electronic device of claim 1 wherein the variable is a global variable that is accessed by, at least, both the processor- executable signal processing instructions and the processor-executable state determination instructions stored in the non-transitory processor-readable storage medium.

10. The wearable electronic device of claim 1 wherein, when executed by the processor, the state determination instructions cause the processor to:

monitor the sensor signals;

enter the wearable electronic device into a sleep state if a period of at least five seconds elapses with no input detected by the at least one sensor, and wherein in the sleep state at least some components of the wearable electronic device power down to conserve energy; and

wake the wearable electronics device out of the sleep state in response to an input detected by the at least one sensor while the wearable electronic device is in the sleep state.

11. A method of operating a wearable electronic device as a state machine to automatically identify and recover from operational errors, the wearable electronic device including at least one sensor responsive to at least one input effected by a user of the wearable electronic device, wherein in response to the at least one input the at least one sensor provides sensor signals, a processor communicatively coupled to the at least one sensor, and a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores at least:

i) processor-executable state determination instructions that include a variable that determines an operational state of the processor, and ii) processor-executable sensor signal processing instructions, wherein the processor executes the state determination instructions to cause the wearable electronic device to perform the method, the method comprising;

automatically identifying when the wearable electronic device encounters an operational error based on a determination that the sensor signal processing instructions are not calibrated; and

automatically recovering from the operational error, wherein automatically recovering from the operational error includes;

in response to the determination that the sensor signal processing instructions are not calibrated, automatically entering the processor into a standby state;

while the processor is in the standby state;

detecting a first indication from the user by the wearable electronic device,

wherein the first indication is a rest gesture performed by the user indicative that the user is ready to calibrate the sensor signal processing instructions; and

recognizing the first indication from the user by the processor;

in response to recognizing the first indication from the user while the processor is in the standby state, automatically entering the processor into a calibration state;

while the processor is in the calibration state;

detecting second indication from the user by the wearable electronic device;

recognizing the second indication from the user by the processor; and

calibrating the sensor signal processing instructions by the processor in response to recognizing the second indication from the user; and

in response to calibrating the sensor signal processing instructions while the processor is in the calibration state, automatically entering the processor into an active state;

while the processor is in the active state;

detecting at least one input by the at least one sensor; and

executing the calibrated sensor signal processing instructions by the processor, wherein executing the calibrated sensor signal processing instructions causes the processor to process the at least one input based on the calibration of the sensor signal processing instructions from the calibration state.

12. The method of claim 11 wherein the at least one sensor is responsive to at least one gesture performed by the user and provides sensor signals in response to the at least one gesture, and wherein the processor-executable sensor signal processing instructions include processor-executable gesture identification instructions, and wherein:

entering the processor into the standby state in response to a determination that the sensor signal processing instructions are not calibrated includes entering the processor into the standby state in response to the determination that the gesture identification instructions are not calibrated;

while the processor is in the standby state, detecting the first indication from the user by the wearable electronic device, the first indication from the user indicative that the user is ready to calibrate the sensor signal processing instructions includes detecting the first indication from the user by the wearable electronic device, the first indication from the user indicative that the user is ready to calibrate the gesture identification instructions;

while the processor is in the calibration state, calibrating the sensor signal processing instructions by the processor in response to recognizing the second indication from the user includes calibrating the gesture identification instructions by the processor in response to recognizing the second indication from the user;

entering the processor into the active state in response to calibrating the sensor signal processing instructions while the processor is in the calibration state includes entering the processor into the active state in response to calibrating the gesture identification instructions while the processor is in the calibration state;

while the processor is in the active state, detecting at least one input by the at least one sensor includes detecting at least one gesture by the at least one sensor; and

while the processor is in the active state, executing the calibrated sensor signal processing instructions by the processor includes executing the calibrated gesture identification instructions, wherein executing the calibrated gesture identification instructions causes the processor to identify the at least one gesture based on the calibration of the gesture identification instructions from the calibration state.

13. The method of claim 12 wherein, while the processor is in the standby state:

detecting the first indication from the user by the wearable electronic device includes detecting, by the wearable electronic device, when the user dons the wearable electronic device; and

recognizing the first indication from the user by the processor includes recognizing, by the processor, when the user dons the wearable electronic device.

14. The method of claim 12 wherein, while the processor is in the calibration state:

detecting the second indication from the user by the wearable electronic device includes detecting, by the at least one sensor, a reference gesture performed by the user;

recognizing the second indication from the user by the processor includes recognizing the reference gesture by the processor; and

calibrating the gesture identification instructions by the processor in response to recognizing the second indication from the user includes calibrating the gesture identification instructions by the processor based on the reference gesture.

15. The method of claim 12 wherein the at least one sensor includes at least one muscle activity sensor selected from the group consisting of:

an electromyography (“

EMG”

) sensor and a mechanomyography (“

MMG”

) sensor; and

wherein;

while the processor is in the active state, detecting at least one gesture by the at least one sensor includes detecting muscle activity by the at least one muscle activity sensor when the user performs the at least one gesture.

16. The method of claim 12 wherein the at least one sensor includes at least one inertial sensor selected from the group consisting of:

an inertial measurement unit, an accelerometer, and a gyroscope; and

wherein;

while the processor is in the active state, detecting at least one gesture by the at least one sensor includes detecting motion of the wearable electronic device by the at least one inertial sensor when the user performs the at least one gesture.

17. The method of claim 11 wherein the wearable electronic device further includes at least one light-emitting diode (“

LED”

) communicatively coupled to the processor, and wherein the method further comprises at least one of;

activating a first color of the at least one LED by the processor in response to entering the standby state;

activating a second color of the at least one LED by the processor in response to entering the calibration state; and

/or

activating a third color of the at least one LED by the processor in response to entering the active state.

18. The method of claim 11 wherein the wearable electronic device further includes a haptic feedback device communicatively coupled to the processor, and wherein the method further comprises at least one of:

activating the haptic feedback device by the processor in response to entering the standby state;

activating the haptic feedback device by the processor in response to entering the calibration state; and

/or

activating the haptic feedback device by the processor in response to entering the active state.

19. The method of claim 11, further comprising:

monitoring the sensor signals by the processor;

in response to the processor monitoring a period of at least five seconds with no input detected by the at least one sensor, entering the wearable electronic device into a sleep state in which at least some components of the wearable electronic device power down to conserve energy; and

in response to an input detected by the at least one sensor while the wearable electronic device is in the sleep state, waking the wearable electronic device out of the sleep state by the processor.