Eye gaze user interface and calibration method

• US 9,557,812 B2
• Filed: 10/29/2013
• Issued: 01/31/2017
• Est. Priority Date: 07/23/2010
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Gregory A. Maltz
• Inventors: Maltz, Gregory A.
• Primary Examiner(s): Spar, Ilana
• Assistant Examiner(s): Rayan, Mihir K
• Application Number:
  US14/066,654
• Publication Number:
  US 20140049452A1
• Time in Patent Office: 1,190 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A software controlled user interface and calibration method for an eye gaze controlled device, designed to accommodate angular accuracy versus time averaging tradeoffs for eye gaze direction sensors. The method can scale between displaying a small to large number of different eye gaze target symbols at any given time, yet still transmit a large array of different symbols to outside devices with minimal user training. At least part of the method may be implemented by way of a virtual window onto the surface of a virtual cylinder, with eye gaze sensitive symbols that can be rotated by eye gaze thus bringing various groups of symbols into view, and then selected by continual gazing. Specific examples of use of this interface and method on an eyeglasses-like head-mountable, vision-controlled, device are disclosed, along with various operation examples including sending and receiving text messages, control of robotic devices and control of remote vehicles.

Claims

1. A method of controlling a vision-controlled, head-mounted frame device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display and at least one eye position tracking sensor, said method comprising:

displaying a plurality of visual targets on said virtual display, said plurality of visual targets each comprising a visible element embedded within an eye position zone with an area that is equal to or larger than said visible element;

at least three or more of said visual targets disposed in a user interface comprising at least a surface of a virtual solid capable of rotation, at least three or more of said plurality of visual targets disposed across said surface in a regularly recurring order, thus providing a predictable disposition of visual elements and corresponding visual targets;

displaying at least three different visible elements from at least three different visual targets from said user interface on said virtual display, producing displayed virtual solid visual targets;

using said at least one eye position tracking sensor to determine when said at least one eye is on average continuously gazing in a direction that is within the eye position zone of at least one of said plurality of visual targets for a time period exceeding a hovering time interval, signaling when said hovering time interval has been exceeded, and if said gaze remains within the eye position zone of said at least one of said plurality of visual targets, without looking away, for a time period exceeding a keypress time interval, registering that an eye gaze controlled virtual key corresponding to said at least one of said displayed virtual solid visual targets has been pressed by said user; and

using said eye gaze controlled virtual key presses to control either transmitting or receiving said information to at least one computerized device outside of said head-mounted frame device.

2. The method of claim 1, wherein said signaling comprises signaling methods selected from the group consisting of changing a visible appearance of said visual target, changing a visible appearance of a different visible portion of said virtual display, creating sounds, or creating inaudible vibrations.

3. The method of claim 1, wherein said virtual solid is invisible, and only the visual targets on said virtual solid are displayed on said virtual display.

4. The method of claim 1, further displaying at least one rotation visual target that is configured so that a keypress on said rotation visual target causes said virtual solid to virtually rotate about a first virtual axis of rotation, thereby suppressing the display of at least one of a previously displayed virtual solid visual targets, and causing at least one new displayed virtual solid visual target to now be displayed on said virtual display.

5. The method of claim 4 wherein said visual targets are disposed in at least one row circling an outer circumference of said virtual solid.

6. The method of claim 5, wherein said visual targets are disposed in more than one row circling an outer circumference of said virtual solid;

and

further displaying at least one shift virtual solid visual target, configured so that a keypress on said shift virtual solid visual target causes said visual targets from a different row of said virtual solid to become displayed virtual solid visual targets.

7. The method of claim 1, wherein said device has at least a standby mode and an active mode, wherein when said device is in a standby mode, although at least some of said area of said visual display will be sensitive to the gazing direction of said at least one eye, more than 50% of the area of said virtual display will not be sensitive to the gazing direction of said at least one eye, thus enabling said user to more freely gaze at outside objects not displayed on said virtual display.

8. The method of claim 7, wherein when said device is in a standby mode, displaying at least one device active visual target with an eye position zone that has an area of less than 50% of the area of said virtual display;

and

when said user presses the virtual key corresponding to said device active visual target, changing a state of said device from standby mode to active mode.

9. The method of claim 1, wherein said virtual targets have visible elements and virtual keys corresponding to a plurality of text symbols selected from the group consisting of the ASCII character set, the extended ASCII character sets, and the Unicode character set.

10. The method of claim 1, wherein said at least some of said virtual targets are assigned to a set of functions selected from the group consisting of data recall functions, data storage functions, device control functions, outside device control functions, user defined macros, and user defined functions.

11. The method of claim 1, wherein at least incoming messages are displayed in a portion of said virtual display that is distinct from the portion of said virtual display that is used for said plurality of visual targets.

12. The method of claim 1, wherein said eye position zone is an approximately circular zone with an approximate virtual diameter, as seen from said at least one eye of said human user, of less than or equal to 10 degrees.

13. The method of claim 1, in which said plurality of visual targets comprises seven or more visual targets.

14. The method of claim 1, wherein said device additionally comprises a camera directed to observe the outside environment, and wherein said virtual display additionally displays augmented reality virtual information.

15. The method of claim 1, wherein said eye position tracking sensor is further calibrated to match the characteristics said at least one eye of said user by the steps of:

instructing said user to gaze at a plurality of visible eye calibration targets on said virtual display, said plurality of visible eye calibration targets having known locations on said virtual display, and determining each time when said at least one eye of said user is gazing at each of said plurality of visible eye calibration targets;

for said each time, using said at least one eye position tracking sensor to obtain eye calibration data pertaining to the appearance of said at least one eye during said each time, and recording said eye calibration data in memory;

and using said eye calibration data as a calibration reference to subsequently determine the eye gaze direction of said at least one eye when said at least one eye is gazing at one or more visual targets on said virtual display.

16. The method of claim 1, wherein said frame comprises frame portions that extend over or behind the ears of said human user, and wherein said frame portions are configured to mount and hold said frame in place on said human user'"'"'s head.

17. The method of claim 1, wherein said eye gaze controlled virtual key is associated with information comprising at least one text symbol, and wherein transmitting said information comprises transmitting said at least one text symbol.