Magnetometer-based gesture sensing with a wearable device

• US 10,146,323 B1
• Filed: 04/13/2017
• Issued: 12/04/2018
• Est. Priority Date: 01/04/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Keyes, Edward, Johnson, Michael Patrick, Starner, Thad Eugene
• Primary Examiner(s): XAVIER, ANTONIO J
• Application Number:
  US15/487,021
• Time in Patent Office: 600 Days
• CPC Class Codes: G01R 33/0029 Treating the measured signals, e.g. removing offset or noise
• CPC Class Codes: G01R 33/0206 Three-component magnetometers
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A wearable computing device such as a head-mounted display (HMD) may be equipped with a magnetometer for detecting presence and motion of a hand-wearable magnet (HWM). The HMD may analyze magnetic field measurements of the magnetometer to determine when the HWM moves within a threshold distance of the magnetometer, and may thereafter determine one or more patterns of motion of the HWM based the magnetic field measurements. The HMD may operate in a background detection state in order to determine a background magnetic field strength and to monitor for magnetic disturbances from the HWM. Upon occurrence of a trigger event corresponding to magnetic disturbance above a threshold level, the HMD may transition to operating in a gesture detection state in which it analyzes magnetometer measurements for correspondence with known gestures. Upon recognizing a known gesture, the HMD may carry out one or more actions based on the recognized known gesture.

Claims

1. A wearable computing device comprising:

a head-wearable component including a magnetometer device with three orthogonal measurement axes;

a remotely-wearable component, different from the head-wearable component, and communicatively coupled with the head-wearable component via at least one of a wired communicative connection or a wireless communicative connection;

one or more processors configured as part of at least one of the head-wearable component or the remotely-wearable component;

memory accessible to the one or more processors, and being configured as part of at least one of the head-wearable component or the remotely-wearable component; and

executable instructions stored in the memory that upon execution by the one or more processors cause the wearable computing device to carry out operations including;

operating in a background detection state,

while operating in the background detection state, carrying out functions of the background state including;

measuring three orthogonal components of a background magnetic field with the magnetometer device, and determining a field magnitude of the background magnetic field from the three measured orthogonal components,

determining an occurrence of a trigger from a hand-wearable magnet (HWM) at a time T_start by detecting a perturbation upon the HWM of the determined field magnitude at least as large as a perturbation threshold,

and upon determining the occurrence of the trigger, transitioning to operating in a gesture detection state, and

while operating in the gesture detection state, carrying out functions of the gesture detection state including;

for the duration of the gesture detection state, tracking motion of the HWM by determining time derivatives of magnetic field strength measured by the magnetometer device along each of the three orthogonal measurement axes,

making a comparison of the determined time derivatives with one or more sets of pre-determined time derivatives of magnetic field strength, wherein each of the one or more sets is stored at the wearable computing device and each is associated with a respective known gesture,

upon matching the determined time derivatives with a particular set of the one or more sets based on the comparison, identifying the respective known gesture associated with the particular set,

and transitioning to operating in the background detection state upon both of, measuring the magnitude perturbation by the HWM of the determined field magnitude to be less than the perturbation threshold, and determining an expiration of a time interval W that begins at T_start.

2. The wearable computing device of claim 1, wherein the operations further include:

responding to identifying the respective known gesture associated with the particular set by identifying a pre-determined computer-executable action associated with the identified respective known gesture; and

carrying out the identified pre-determined computer-executable action with the one or more processors.

3. The wearable computing device of claim 1, wherein measuring the three orthogonal components of a background magnetic field with the magnetometer device comprises measuring magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=N₁, . . . , N₂, over a sliding time window W_N1,N2 from N₁ to N₂, wherein N₂=N₁+N−

1, and N≥

1,

wherein determining the field magnitude of the background magnetic field from the three measured orthogonal components comprises;

computing magnetic field magnitude samples h_i=∥

H∥

=√

{square root over (H_x²+H_y²+H_z²)} at the consecutive discrete times t_i, i=N₁, . . . , N₂ over W_N1,N2; and

computing a sample mean field strength

4. The wearable computing device of claim 3, wherein the wearable computing device further includes a motion detector,

wherein making the determination that the at least one of the field magnitude samples h_j deviates from μ

_N1,N2 by an amount at least as large as the perturbation threshold comprises both of;

determining that the at least one of the field magnitude samples h_j deviates from μ

_N1,N2 by an amount at least as large as the perturbation threshold; and

concurrently determining that the motion detector is detecting motion at no greater than a threshold level of motion.

5. The wearable computing device of claim 1, wherein each of the one or more sets of pre-determined time derivatives of magnetic field strength comprises a respective triplet of pre-determined sequences of discrete time derivatives of a magnetic field along the three orthogonal measurement axes,

wherein determining the time derivatives of magnetic field strength measured by the magnetometer device along each of the three orthogonal measurement axes comprises:

during at least a portion of the time interval W, measuring orthogonal magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=1, . . . , N, wherein N≥

2, to obtain samples of the of H_x, H_y, and H_z at the consecutive discrete times t_i, i=1, . . . , N; and

computing discrete time derivatives of the measured orthogonal magnetic field components H_x, H_y, and H_z from discrete differences between successive samples,

and wherein matching the determined time derivatives with the particular set of the one or more sets based on the comparison comprises determining a closest match between the computed discrete time derivatives and the respective triplet of pre-determined sequences of discrete time derivatives of one of the one or more sets.

6. The wearable computing device of claim 1, wherein each of the one or more sets of pre-determined time derivatives of magnetic field strength comprises a respective triplet of pre-determined sequences of discrete time derivatives of a magnetic field along the three orthogonal measurement axes,

and wherein the operations further include:

transitioning to operating in a gesture recording state in response to receiving a recording command; and

while operating in the gesture recording state, carrying out functions of the gesture recording state including,

determining an occurrence of a recording trigger from the HWM by detecting a perturbation by the HWM of the determined field magnitude at least as large as a recording perturbation threshold,

upon determining the occurrence of the recording trigger, measuring orthogonal magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=1, . . . , N, wherein N≥

2, to obtain samples of the of H_x, H_y, and H_z at the consecutive discrete times t_i, i=1, . . . ,

computing discrete time derivatives of the measured orthogonal magnetic field components H_x, H_y, and H_z from discrete differences between successive samples to obtain a recorded triplet of sequences of discrete time derivatives

7. The wearable computing device of claim 1, wherein the three orthogonal measurement axes comprise a first triplet of orthogonal measurement axes of the magnetometer device,

wherein the magnetometer device comprises a second triplet of orthogonal measurement axes,

wherein operating in the background detection state further comprises carrying out further functions of the background detection state including:

measuring three orthogonal components of the background magnetic field with the second triplet of orthogonal axes to determine a supplementary field magnitude of the background magnetic field,

determining an occurrence of a supplementary trigger from the HWM by detecting a supplementary perturbation by the HWM of the determined supplementary field magnitude at least as large as a supplementary perturbation threshold,

and upon determining the occurrence of at least one of the trigger and the supplementary trigger, transitioning to operating in the gesture detection state;

and wherein operating in the gesture detection state further comprises carrying out further functions of the gesture detection state including;

tracking motion of the HWM by determining supplementary time derivatives of magnetic field strength measured with the second triplet of orthogonal axes,

making a joint comparison of the determined time derivatives and the determined supplementary time derivatives with one or more combined sets of pre-determined time derivatives and pre-determined supplementary time derivatives, each of the one or more combined sets being stored at the wearable HMD and each being associated with one of the respective known gestures,

upon jointly matching both the determined time derivatives and the determined supplementary time derivatives with a particular combined set of the one or more combined sets based on the joint comparison, identifying the respective known gesture associated with the particular combined set.

8. The wearable computing device of claim 1, wherein the head-wearable component further includes a user input/output component, the user input/output component being at least one of:

one or more speakers, one or more earphones, a still-image capture device, or a video-image capture device.

9. In a wearable computing device comprising a head-wearable component including a magnetometer device with three orthogonal measurement axes, a computer-implemented method comprising:

operating the wearable computing device in a background detection state, wherein the wearable computing device further comprises a remotely-wearable component, different from the head-wearable component, communicatively coupled with the head-wearable component via at least one of a wired communicative connection or a wireless communicative connection;

while operating in the background detection state, carrying out functions of the background state including;

measuring three orthogonal components of a background magnetic field with the magnetometer device, and determining a field magnitude of the background magnetic field from the three measured orthogonal components,

determining an occurrence of a trigger from a hand-wearable magnet (HWM) at a time T_start by detecting a perturbation upon the HWM of the determined field magnitude at least as large as a perturbation threshold,

and upon determining the occurrence of the trigger, transitioning to operating in a gesture detection state, and

while operating in the gesture detection state, carrying out functions of the gesture detection state including;

for the duration of the gesture detection state, tracking motion of the HWM by determining time derivatives of magnetic field strength measured by the magnetometer device along each of the three orthogonal measurement axes,

making a comparison of the determined time derivatives with one or more sets of pre-determined time derivatives of magnetic field strength, wherein each of the one or more sets is stored at the wearable computing device and each is associated with a respective known gesture,

upon matching the determined time derivatives with a particular set of the one or more sets based on the comparison, identifying the respective known gesture associated with the particular set,

and transitioning to operating in the background detection state upon both of, measuring the magnitude perturbation by the HWM of the determined field magnitude to be less than the perturbation threshold, and determining an expiration of a time interval W that begins at T_start.

10. The method of claim 9, further comprising:

responding to identifying the respective known gesture associated with the particular set by identifying a pre-determined computer-executable action associated with the identified respective known gesture; and

carrying out the identified pre-determined computer-executable action with the one or more processors.

11. The method of claim 9, wherein measuring the three orthogonal components of a background magnetic field with the magnetometer device comprises measuring magnetic field components H_x, H_y, and H_z of a magnetic field vector H=[H_x, H_y, H_z] at consecutive discrete times t_i, i=N₁, . . . , N₂, over a sliding time window W_N1,N2 from N₁ to N₂, wherein N₂=N₁+N−

1, and N≥

1,

wherein determining the field magnitude of the background magnetic field from the three measured orthogonal components comprises;

computing magnetic field magnitude samples h_i=∥

H∥

=√

{square root over (H_x²+H_y²+H_z²)} at the consecutive discrete times t_i, i=N₁, . . . , N₂ over W_N1,N2; and

computing a sample mean field strength

12. The method of claim 11, wherein the wearable computing device further includes a motion detector,

wherein making the determination that the at least one of the field magnitude samples h_j deviates from μ

_N1,N2 by an amount at least as large as the perturbation threshold comprises both of;

determining that the at least one of the field magnitude samples h_j deviates from μ

_N1,N2 by an amount at least as large as the perturbation threshold; and

concurrently determining that the motion detector is detecting motion at no greater than a threshold level of motion.

13. The method of claim 9, wherein each of the one or more sets of pre-determined time derivatives of magnetic field strength comprises a respective triplet of pre-determined sequences of discrete time derivatives of a magnetic field along the three orthogonal measurement axes,

wherein determining the time derivatives of magnetic field strength measured by the magnetometer device along each of the three orthogonal measurement axes comprises:

during at least a portion of the time interval W, measuring orthogonal magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=1, . . . , N, wherein N≥

2, to obtain samples of the of H_x, H_y, and H_z at the consecutive discrete times t_i, i=1, . . . , N; and

computing discrete time derivatives of the measured orthogonal magnetic field components H_x, H_y, and H_z from discrete differences between successive samples,

and wherein matching the determined time derivatives with the particular set of the one or more sets based on the comparison comprises determining a closest match between the computed discrete time derivatives and the respective triplet of pre-determined sequences of discrete time derivatives of one of the one or more sets.

14. The method of claim 9, wherein each of the one or more sets of pre-determined time derivatives of magnetic field strength comprises a respective triplet of pre-determined sequences of discrete time derivatives of a magnetic field along the three orthogonal measurement axes,

and wherein the method further comprises:

transitioning to operating in a gesture recording state in response to receiving a recording command; and

while operating in the gesture recording state, carrying out functions of the gesture recording state including,

determining an occurrence of a recording trigger from the HWM by detecting a perturbation by the HWM of the determined field magnitude at least as large as a recording perturbation threshold,

upon determining the occurrence of the recording trigger, measuring orthogonal magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=1, . . . , N, wherein N≥

2, to obtain samples of the of H_x, H_y, and H_z at the consecutive discrete times t_i, i=1, . . . , N,

computing discrete time derivatives of the measured orthogonal magnetic field components H_x, H_y, and H_z from discrete differences between successive samples to obtain a recorded triplet of sequences of discrete time derivatives

15. The method of claim 9, wherein the three orthogonal measurement axes comprise a first triplet of orthogonal measurement axes of the magnetometer device,

wherein the magnetometer device comprises a second triplet of orthogonal measurement axes,

wherein operating in the background detection state further comprises carrying out further functions of the background detection state including:

measuring three orthogonal components of the background magnetic field with the second triplet of orthogonal axes to determine a supplementary field magnitude of the background magnetic field,

determining an occurrence of a supplementary trigger from the HWM by detecting a supplementary perturbation by the HWM of the determined supplementary field magnitude at least as large as a supplementary perturbation threshold,

and upon determining the occurrence of at least one of the trigger and the supplementary trigger, transitioning to operating in the gesture detection state;

and wherein operating in the gesture detection state further comprises carrying out further functions of the gesture detection state including;

tracking motion of the HWM by determining supplementary time derivatives of magnetic field strength measured with the second triplet of orthogonal axes,

making a joint comparison of the determined time derivatives and the determined supplementary time derivatives with one or more combined sets of pre-determined time derivatives and pre-determined supplementary time derivatives, each of the one or more combined sets being stored at the wearable HMD and each being associated with one of the respective known gestures,

upon jointly matching both the determined time derivatives and the determined supplementary time derivatives with a particular combined set of the one or more combined sets based on the joint comparison, identifying the respective known gesture associated with the particular combined set.

16. The method of claim 9, wherein the head-wearable component further includes a user input/output component, the user input/output component being at least one of:

one or more speakers, one or more earphones, a still-image capture device, or a video-image capture device.

17. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable computing device, cause the wearable computing device to carry out operations comprising:

operating the wearable computing device in a background detection state, wherein the wearable computing device comprises a head-wearable component and a remotely-wearable component, different from the head-wearable component, communicatively coupled with the head-wearable component via at least one of a wired communicative connection or a wireless communicative connection;

while operating in the background detection state, carrying out functions of the background state including,

measuring three orthogonal components of a background magnetic field using three orthogonal measurement axes of a magnetometer device of the head-wearable component, and determining a field magnitude of the background magnetic field from the three measured orthogonal components,

determining an occurrence of a trigger from a hand-wearable magnet (HWM) at a time T_start upon detecting a perturbation by the HWM of the determined field magnitude at least as large as a perturbation threshold,

and upon determining the occurrence of the trigger, transitioning to operating in a gesture detection state; and

while operating in the gesture detection state, carrying out functions of the gesture detection state including,

for the duration of the gesture detection state, tracking motion of the HWM by determining time derivatives of magnetic field strength measured by the magnetometer device along each of the three orthogonal measurement axes,

making a comparison of the determined time derivatives with one or more sets of pre-determined time derivatives of magnetic field strength, wherein each of the one or more sets is configured to be stored at the wearable HMD and each is associated with a respective known gesture,

upon matching the determined time derivatives with a particular set of the one or more sets based on the comparison, identifying the respective known gesture associated with the particular set,

and transitioning to operating in the background detection state upon both of, measuring the magnitude perturbation by the HWM of the determined field magnitude to be less than the perturbation threshold, and determining an expiration of a time interval W that begins at T_start.

18. The nontransitory computer-readable medium of claim 17, wherein the operations further include:

responding to identifying the respective known gesture associated with the particular set by identifying a pre-determined computer-executable action associated with the identified respective known gesture; and

carrying out the identified pre-determined computer-executable action with the one or more processors.

19. The nontransitory computer-readable medium of claim 17, wherein each of the one or more sets of pre-determined time derivatives of magnetic field strength comprises a respective triplet of pre-determined sequences of discrete time derivatives of a magnetic field along the three orthogonal measurement axes,

and wherein the operations further include:

transitioning to operating in a gesture recording state in response to receiving a recording command; and

while operating in the gesture recording state, carrying out functions of the gesture recording state including,

determining an occurrence of a recording trigger from the HWM by detecting a perturbation by the HWM of the determined field magnitude at least as large as a recording perturbation threshold,

upon determining the occurrence of the recording trigger, measuring orthogonal magnetic field components H_x, H_y, and H_z of a magnetic field vector {right arrow over (H)}=[H_x, H_y, H_z] at consecutive discrete times t_i, i=1, . . . , N, wherein N≥

2, to obtain samples of the of H_x, H_y, and H_z at the consecutive discrete times t_i, i=1, . . . , N,

computing discrete time derivatives of the measured orthogonal magnetic field components H_x, H_y, and H_z from discrete differences between successive samples to obtain a recorded triplet of sequences of discrete time derivatives

20. The nontransitory computer-readable medium of claim 17, wherein the head-wearable component further includes a user input/output component, the user input/output component being at least one of:

one or more speakers, one or more earphones, a still-image capture device, or a video-image capture device,

and wherein the nontransitory computer-readable medium further stores executable instructions for operating the input/output component by the one or more processors.