Input detection

• US 9,116,545 B1
• Filed: 03/21/2012
• Issued: 08/25/2015
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Raffle, Hayes Solos, Biffle, Cliff L.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): McLoone, Peter D
• Application Number:
  US13/425,830
• Time in Patent Office: 1,252 Days
• Field of Search: 345/8, 345/156, 351209-210, 348/E7.085
• US Class Current: 1/1
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Example methods and systems determine viewing states, blinks, and blink intervals of an eye of a wearer of a head-mountable device. The head-mountable display can emit IR radiation from an associated IR radiation source toward a target location. An IR sensor associated with the head-mountable display can receive reflected IR radiation, such as the IR radiation emitted by the IR radiation source and reflected from the target location. The IR sensor can generate amplitude data for the reflected IR radiation. The head-mountable display can be used to determine a viewing state of the target location. The viewing state can be based on the amplitude data. The viewing state can determined from among a no-wearer-present viewing state, a wearer-present viewing state, a closed-eye viewing state, an open-eye viewing state, a non-display-viewing viewing state, and a display-viewing viewing state.

Claims

1. A method, comprising:

initiating, by a head-mountable device, a camera application;

receiving, by the head-mountable device, first audio data that is generated by an audio sensor associated with the head-mountable device;

detecting that the first audio data includes a speech command that corresponds to a video recording process;

in response to the speech command, initiating the video recording process via the camera application, wherein the video recording process comprises capturing second audio data and video data and storing the second audio data and the video data to a video file;

emitting infrared (IR) radiation from an IR radiation source associated with the head-mountable device toward a target location;

receiving, at an IR sensor associated with the head-mountable device, reflected IR radiation, wherein the reflected IR radiation comprises IR radiation emitted by the IR radiation source and reflected from the target location;

generating time-varying amplitude data for the reflected IR radiation received at the IR sensor;

determining, based on the time-varying amplitude data, a sequence of viewing states of the target location; and

during the video recording process;

(a) detecting, in the sequence of viewing states, a portion of the sequence corresponding to an eye closure of a first duration;

(b) determining that the first duration is within a first range of durations corresponding to a voluntary blink, wherein the durations in the first range are longer than durations in a second range of durations corresponding to an involuntary blink; and

(c) in response to determining that the first duration is within the first range of durations corresponding to a voluntary blink, ceasing the capturing of second audio data and continuing the capturing and storing of video data to the video file.

2. The method of claim 1, wherein determining the sequence of viewing states of the target location comprises:

comparing the time-varying amplitude data to calibration data corresponding to a plurality of viewing states.

3. The method of claim 1, further comprising:

controlling the head-mountable device based on the sequence of viewing states.

4. The method of claim 3, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein controlling the head-mountable device based on the sequence of viewing states comprises:

in response to the second viewing state differing from the first viewing state, generating a command for the head-mountable device.

5. The method of claim 3, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein controlling the head-mountable device based on the sequence of viewing states comprises:

in response to the second viewing state equaling the first viewing state, generating a command for the head-mountable device.

6. An article of manufacture including a non-transitory computer-readable medium having instructions stored thereon that, upon execution of the instructions by a computing device, cause the computing device to perform functions comprising:

initiating a camera application associated with a head-mountable device;

receiving first audio data that is generated by an audio sensor associated with the head-mountable device;

detecting that the first audio data includes a speech command that corresponds to a video recording process;

in response to the speech command, initiating the video recording process via the camera application, wherein the video recording process comprises capturing second audio data and video data and storing the second audio data and the video data to a video file;

emitting infrared (IR) radiation from an IR radiation source associated with the head-mountable device toward a target location;

receiving, at an IR sensor associated with the head-mountable device, reflected IR radiation, wherein the reflected IR radiation comprises IR radiation emitted by the IR radiation source and reflected from the target location;

generating time-varying amplitude data for the reflected IR radiation;

determining, based on the time-varying amplitude data, a sequence of viewing states of the target location; and

during the video recording process;

(a) detecting, in the sequence of viewing states, a portion of the sequence corresponding to an eye closure of a first duration;

(b) determining that the first duration is within a first range of durations corresponding to a voluntary blink, wherein the durations in the first range are longer than durations in a second range of durations corresponding to an involuntary blink; and

(c) in response to determining that the first duration is within the first range of durations corresponding to a voluntary blink, ceasing the capturing of second audio data and continuing the capturing and storing of video data to the video file.

7. The article of manufacture of claim 6, wherein determining the sequence of viewing states of the target location comprises:

comparing the time-varying amplitude data to calibration data corresponding to a plurality of viewing states.

8. The article of manufacture of claim 6, further comprising:

controlling the head-mountable device based on the sequence of viewing states.

9. The article of manufacture of claim 8, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein controlling the head-mountable device based on the sequence of viewing states comprises:

in response to the second viewing state differing from the first viewing state, generating a command for the head-mountable device.

10. The article of manufacture of claim 8, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein controlling the head-mountable device based on the sequence of viewing states comprises:

in response to the second viewing state equaling the first viewing state, generating a command for the head-mountable device.

11. A head-mountable device, comprising:

a head-mountable display;

an audio sensor;

an infrared (IR) radiation source, configured to emit IR radiation toward a target location;

an IR sensor configured to receive IR radiation reflected from the target location and to generate time-varying amplitude data for the reflected IR radiation; and

a processor, configured to;

initiate a camera application;

receive first audio data that is generated by the audio sensor;

detect that the first audio data includes a speech command that corresponds to a video recording process;

in response to the speech command, initiate the video recording process via the camera application, wherein the video recording process comprises capturing second audio data and video data and storing the second audio data and the video data to a video file;

receive the time-varying amplitude data;

determine, based on the time-varying amplitude data, a sequence of viewing states of the target location; and

during the video recording process;

(a) detect, in the sequence of viewing states, a portion of the sequence corresponding to an eye closure of a first duration;

(b) determine that the first duration is within a first range of durations corresponding to a voluntary blink, wherein the durations in the first range are longer than durations in a second range of durations corresponding to an involuntary blink; and

(c) in response to determining that the first duration is within the first range of durations corresponding to a voluntary blink, cease the capturing of second audio data and continuing the capturing and storing of video data to the video file.

12. The head-mountable device of claim 11, wherein the processor is configured to determine the sequence of viewing states of the target location by at least:

comparing the time-varying amplitude data to calibration data corresponding to a plurality of viewing states.

13. The head-mountable device of claim 11, wherein the processor is further configured to:

control the head-mountable display based on the sequence of viewing states.

14. The head-mountable device of claim 13, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein the processor is configured to control the head-mountable display based on the sequence of viewing states by generating a command for the head-mountable display in response to the second viewing state differing from the first viewing state.

15. The head-mountable device of claim 13, wherein the sequence of viewing states comprises a first viewing state at a first time and a second viewing state at a second time, wherein the second time is after the first time, and wherein the processor is configured to control the head-mountable display based on the sequence of viewing states by generating a command for the head-mountable display in response to the second viewing state equaling the first viewing state.

16. The head-mountable device of claim 11, further comprising an optical system, the optical system comprising the IR radiation source and a display light source configured to emit visible light toward the target location and wherein the optical system is configured so that the IR radiation and the visible light follow a common path to the target location.

17. The head-mountable device of claim 16, wherein the IR sensor is configured to be physically separate from the optical system.

18. The head-mountable device of claim 11, wherein the sequence of viewing states of the target location comprises viewing states from among a no-wearer-present viewing state, a wearer-present viewing state, a closed-eye viewing state, an open-eye viewing state, a non-display-viewing viewing state, and a display-viewing viewing state.