EYE GAZE USER INTERFACE AND CALIBRATION METHOD

• US 20140049452A1
• Filed: 10/29/2013
• Published: 02/20/2014
• Est. Priority Date: 07/23/2010
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: TelepathEye, Inc.
• Inventors: Maltz, Gregory A.
• Granted Patent: US 9,557,812 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A software controlled user interface and calibration method for an eye gaze controlled device, designed to accommodate angular accuracy versus time averaging tradeoffs for eye gaze direction sensors. The method can scale between displaying a small to large number of different eye gaze target symbols at any given time, yet still transmit a large array of different symbols to outside devices with minimal user training. At least part of the method may be implemented by way of a virtual window onto the surface of a virtual cylinder, with eye gaze sensitive symbols that can be rotated by eye gaze thus bringing various groups of symbols into view, and then selected by continual gazing. Specific examples of use of this interface and method on an eyeglasses-like head-mountable, vision-controlled, device are disclosed, along with various operation examples including sending and receiving text messages, control of robotic devices and control of remote vehicles.

Claims

1. A method of controlling a head-mountable, vision-controlled, device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display and at least one eye position tracking sensor, method comprising:

displaying a plurality of visual targets on said virtual display, said plurality of visual targets each comprising a visible element embedded within a eye position zone with an area that is equal to or larger than said visible element;

using said at least one eye position tracking sensor to determine when said at least one eye is on average gazing in a direction that is within the eye position zone of at least one of said plurality of visible elements for a time period exceeding a hovering time interval, signaling when said hovering time interval has been exceeded, and if said gaze remains within the eye position zone of said at least one of said plurality of visible elements for a time period exceeding a keypress time interval, registering that a virtual key corresponding to said at least one of said plurality of visible targets has been pressed by said user; and

using said virtual key presses to control either transmitting or receiving said information.

2. The method of claim 1, wherein said signaling comprises signaling methods selected from the group consisting of changing the visible appearance of said visual target, changing the visible appearance of a different visible portion of said virtual display, creating sounds, or creating inaudible vibrations.

3. The method of claim 1, further creating a user interface comprising at least a surface of a virtual solid, and disposing at least three or more of said plurality of visual targets across said surface;

displaying at least three different visible elements from at least three different visual targets from said user interface on said virtual display, producing displayed virtual solid visual targets;

and determining when a virtual key corresponding to said at least one of said displayed virtual solid visual targets has been pressed by said user.

4. The method of claim 3, wherein said virtual solid is invisible, and only the visual targets on said virtual solid are displayed on said virtual display.

5. The method of claim 3, further displaying at least one rotation visual target that is configured so that a keypress on said rotation visual target causes said virtual solid to virtually rotate about a first virtual axis of rotation, thereby suppressing the display of at least one of said previously displayed virtual solid visual targets, and causing at least one new displayed virtual solid visual target to now be displayed on said virtual display.

6. The method of claim 5 wherein said visual targets are disposed in at least one row circling the outer circumference of said virtual solid.

7. The method of claim 6, wherein said visual targets are disposed in more than one row circling the outer circumference of said virtual solid;

and

further displaying at least one shift virtual solid visual target, configured so that a keypress on said shift virtual solid visual target causes said visual targets from a different row of said virtual solid to become displayed virtual solid visual targets.

8. The method of claim 1, wherein said device has at least a standby mode and an active mode, wherein when said device is in a standby mode, more than 50% of the area of said virtual display will not be sensitive to the gazing direction of said at least one eye, thus enabling said user to more freely gaze at outside objects not displayed on said virtual display.

9. The method of claim 8, wherein when said device is in a standby mode, displaying at least one device active visual target with a eye position zone that has an area of less than 50% of the area of said virtual display;

and

when said user presses the virtual key corresponding to said device active visual target, changing the state of said device from standby mode to active mode.

10. The method of claim 1, wherein said virtual targets have visible elements and virtual keys corresponding to a plurality of symbols selected from the group consisting of the ASCII character set, the extended ASCII character sets, and the Unicode character set.

11. The method of claim 1, wherein said at least some of said virtual targets are assigned to a set of functions selected from the group consisting of caller directory functions, data recall functions, data storage functions, device control functions, outside device control functions, user defined macros, and user defined functions.

12. The method of claim 1, wherein one or more of either incoming messages or outgoing messages are displayed in a portion of said virtual display is distinct from the portion of said virtual display that is used for said plurality of visual targets.

13. The method of claim 1, wherein said eye position zone is an approximately circular zone with an approximate virtual diameter, as seen from said at least one eye of said human user, of less than or equal to 10 degrees.

14. The method of claim 1, in which said plurality of visual targets comprises seven or more visual targets.

15. The method of claim 1, wherein said device additionally comprises a camera directed to observe the outside environment, and wherein said virtual display additionally displays augmented reality virtual information.

16. The method of claim 1, wherein said eye position tracking sensor is further calibrated to match the characteristics said at least one eye of said user by the steps of:

instructing said user to gaze at a plurality of visible eye calibration targets on said virtual display, said plurality of visual eye calibration targets having known locations on said virtual display, and determining each time when said at least one eye of said user is gazing at each of said plurality of visual eye calibration targets;

for said each time, using said at least one eye position tracking sensor to obtain eye calibration data pertaining to the appearance of said at least one eye during said each time, and recording said eye calibration data in memory;

and using said eye calibration data as a calibration reference to subsequently determine the eye gaze direction of said at least one eye when said at least one eye is gazing at one or more visual targets on said virtual display.

17. A method of calibrating the eye position tracking sensor of a head-mountable, vision-controlled, device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display and at least one eye position tracking sensor, said method comprising:

displaying, over a plurality of time sequences, different visual eye calibration targets on said virtual display, each different visual eye calibration target being displayed at known location on said virtual display, and each comprising a visible element embedded within an eye position zone with an area that is equal to or larger than said visible element;

for at least some of said time sequences, instructing said user to gaze at said different visual eye calibration targets, and determining each time when said at least one eye of said user is gazing at each of said plurality of visual eye calibration targets;

for said each time, using said at least one eye position tracking sensor to obtain eye calibration data pertaining to the appearance of said at least one eye during said each time, and recording said eye calibration data in memory;

and using said eye calibration data as a calibration reference to subsequently determine the eye gaze direction of said at least one eye when said at least one eye is gazing at one or more visual targets on said virtual display.

18. The method of claim 17, wherein said eye position tracking sensor is a video camera configured to acquire video images of said at least one eye;

and wherein said eye calibration data comprises at least one image of said at least one eye.

19. The method of claim 17, further illuminating said at least one eye of said user with a plurality of light sources comprising different light illumination angles and different light illumination wavelengths during said calibration.

20. The method of claim 17, further using said eye calibration data, at least one processor, and pattern recognition software to subsequently determine the eye gaze direction of said at least one eye when said at least one eye is gazing at one or more visual targets on said virtual display.