EYE-WEARABLE DEVICE USER INTERFACE AND AUGMENTED REALITY METHOD

• US 20160004306A1
• Filed: 07/13/2015
• Published: 01/07/2016
• Est. Priority Date: 07/23/2010
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: TelepathEye, Inc.
• Inventors: Maltz, Gregory A.
• Granted Patent: US 9,977,496 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user

Abstract

A software controlled user interface and method for a head-mountable device equipped with at least one display or connectivity to at least one touch pad. The method can scale between displaying a small to large number of different eye gaze target symbols at any given time, yet still transmit a large array of different symbols to outside devices. At least part of the method may be implemented by way of a virtual window onto the surface of a virtual cylinder, with touchpad or eye gaze sensitive symbols that can be rotated by touchpad touch or eye gaze thus bringing various groups of symbols into view, and then selected. Specific examples of use of this interface and method on a head-mountable device are disclosed, along with various operation examples including transmitting data including text, functionality in virtual or augmented reality, control of robotic devices and control of remote vehicles.

Claims

1. A method of controlling a head-mountable device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display and connectivity to at least one user input device, wherein said at least one user input device comprises any of a touchpad or at least one eye position tracking sensor;

said device comprising a frame configured to attach to the head of a user;

said method comprising;

using at least one processor to display a plurality of visual targets on said virtual display, said plurality of visual targets each comprising a visual element area embedded within a visual element position zone with an area that is equal to or larger than said visual element, each visual element area and visual element position zone mapping to a touch sensitive touch element on a touchpad position zone;

using said at least one user input device to register that a virtual key corresponding to said at least one of said plurality of visual targets has been selected by said user, thereby creating a selected virtual key;

using at least one user input device to register that said selected virtual key has been accepted by said user, thereby creating virtual key activation;

using said virtual key activation to control either transmitting or receiving said information.

2. The method of claim 1, further using said at least one user input device to determine when at least one user is selecting an area that is within a visual element position zone of at least one of said visual element areas, and signaling said selection, prior to using said at least one user input device to register that a virtual key corresponding to said at least one of said plurality of visual targets has been activated by said user.

3. The method of claim 2, wherein said signaling comprises signaling methods selected from the group consisting of changing a visual appearance of said visual target, changing a visual appearance of a different visual portion of said virtual display, creating sounds, or creating inaudible or tactile vibrations.

4. The method of claim 1, further creating a user interface comprising at least a surface of a virtual solid, and disposing at least three or more of said plurality of visual targets across said surface;

displaying at least three different visual elements from at least three different visual targets from said user interface on said virtual display, producing displayed virtual solid visual targets; and

determining when a virtual key corresponding to said at least one of said displayed virtual solid visual targets has been selected and activated by said user.

5. The method of claim 4, further displaying at least one rotation visual target that is configured so that a user selection and activation on said rotation visual target causes said virtual solid to virtually rotate about a first virtual axis of rotation, thereby suppressing display of at least one previously displayed virtual solid visual targets, and causing at least one new displayed virtual solid visual target to now be displayed on said virtual display.

6. The method of claim 5 wherein said visual targets are disposed in at least one row circling an outer circumference of said virtual solid.

7. The method of claim 6, wherein said visual targets are disposed in more than one row circling said outer circumference of said virtual solid;

and

further displaying at least one shift virtual solid visual target, configured so that a user selection on said shift virtual solid visual target causes said visual targets from a different row of said virtual solid to become displayed virtual solid visual targets.

8. The method of claim 5, wherein said least one user input device comprises at least one touchpad, said at least one touchpad comprising a multi-touch touchpad configured to recognize simultaneous touches on at least two different portions of a same touchpad to further control the display of said rotation visual target to at least rotate about said first virtual axis of rotation, or to control an apparent size of said virtual target on said virtual display by any of a swiping, pinching, or double-tapping process.

9. The method of claim 1, wherein said a head-mountable, vision-controlled, device further comprises at least one eye position tracking sensor and said least one user input device comprises at least one eye position tracking sensor;

wherein at least some of said plurality of visual targets further comprise a visual element embedded within an eye position zone with an area that is equal to or larger than said visual element;

further using said at least one eye position tracking sensor to determine when said at least one eye is on average gazing in a direction that is within the eye position zone of at least one of said visual elements, thereby selecting at least one of said visual elements.

10. The method of claim 1, wherein said device further comprises at least one camera disposed to view the outside world, and connectivity to a computer vision system;

further acquiring images from said at least one camera, and using said computer vision system to recognize when said at least one camera is viewing an object, thereby producing an object recognition signal; and

using said object recognition signal to control display of at least some visual targets on said virtual display.

11. The method of claim 10, wherein said touchpad is a virtual touchpad that is displayed by said virtual display, and wherein said at least one camera and computer vision system is configured to recognize when a user is touching said virtual touchpad by imaging the position of a user hand, fingers, or other user controlled pointing device in a space observed by said camera.

12. A method of controlling a head-mountable device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display and connectivity to at least one user input device, wherein said at least one user input device comprises both an touchpad and at least one eye position tracking sensor;

said device comprising a frame configured to attach to the head of a user;

said method comprising;

using at least one processor to display a plurality of visual targets on said virtual display, said plurality of visual targets each comprising a visual element area embedded within a visual element position zone with an area that is equal to or larger than said visual element, each visual element area and visual element position zone mapping to a touch sensitive touch element on a touchpad position zone;

further using said at least one eye position tracking sensor to determine when said at least one eye is on average gazing in a direction that is within said visual element position zone of at least one of said visual elements, thereby selecting at least one of said visual elements;

further signaling said selection, prior to using said at least one user input device to register that a virtual key corresponding to said at least one of said plurality of visual targets has been activated by said user;

using said at least one user input device to register that a virtual key corresponding to said at least one of said plurality of visual targets has been selected by said user, thereby creating a selected virtual key;

using at least one user input device to register that said selected virtual key has been accepted by said user, thereby creating virtual key activation; and

using said virtual key activation to control either transmitting or receiving said information.

13. The method of claim 12, wherein said signaling comprises signaling methods selected from the group consisting of changing a visual appearance of said visual target, changing a visual appearance of a different visual portion of said virtual display, creating sounds, or creating inaudible or tactile vibrations;

and

using said at least one touchpad to register that a virtual key corresponding to said at least one of said plurality of visual targets has been activated by said user.

14. The method of claim 12, further creating a user interface comprising at least a surface of a virtual solid, and disposing at least three or more of said plurality of visual targets across said surface;

displaying at least three different visual elements from at least three different visual targets from said user interface on said virtual display, producing displayed virtual solid visual targets; and

determining when a virtual key corresponding to said at least one of said displayed virtual solid visual targets has been selected and activated by said user;

further displaying at least one rotation visual target that is configured so that a user selection and activation of said rotation visual target causes said virtual solid to virtually rotate about a first virtual axis of rotation, thereby suppressing display of at least one previously displayed virtual solid visual targets, and causing at least one new displayed virtual solid visual target to now be displayed on said virtual display;

wherein said visual targets are disposed in at least one row circling an outer circumference of said virtual solid.

15. The method of claim 14, wherein said at least one touchpad is a multi-touch touchpad configured to recognize simultaneous touches on at least two different portions of a same touchpad to further control the display of said rotation visual target to at least rotate about said first virtual axis of rotation, or to control an apparent size of said virtual target on said virtual display by any of a swiping, pinching, or double-tapping process.

16. The method of claim 12, wherein said device further comprises at least one camera disposed to view the outside world, and connectivity to a computer vision system;

further acquiring images from said at least one camera, and using said computer vision system to recognize when said at least one camera is viewing a visual object, thereby producing an object recognition signal; and

using said object recognition signal to control the display of at least some visual targets on said virtual display.

17. The method of claim 16, wherein said touchpad is a virtual touchpad that is displayed by said virtual display, and wherein said at least one camera and computer vision system is configured to recognize when a user is touching said virtual touchpad by imaging the position of a user hand, fingers, or other user controlled pointing device in a space observed by said camera.

18. A head-mountable device for transmitting and receiving information from a human user with at least one eye, said device having a virtual display, at least one processor, and connectivity to at least one touchpad;

said device further comprising a frame configured to attach to the head of a user;

wherein said connectivity to at least one touchpad is achieved by either a touchpad that is physically attached to said frame, thereby forming a head mounted touchpad, or wherein connectivity to at least one touchpad comprises a wireless or wired connection to at least one touchpad not physically attached to said frame;

said device further comprising;

at least one processor configured to display a plurality of visual targets on said virtual display, said plurality of visual targets each comprising a visual element area embedded within a visual element position zone with an area that is equal to or larger than said visual element each visual element area and visual element position zone mapping to a touch sensitive touch element on a touchpad position zone;

said at least one processor configured to accept input from at least one touchpad to determine when at least one user touch is on average touching an area that selects a visual element position zone of at least one of said visual element;

said at least one processor further configured to accept input from said at least one touchpad to register that a virtual key corresponding to said at least one of said plurality of visual targets has been activated by said user;

said at least one processor further configured to use said virtual key activation to control either transmitting or receiving said information.

19. The device of claim 18, wherein said device further comprises at least one eye position tracking sensor;

said at least one processor is configured to associate at least some of said plurality of visual targets with a visual element embedded within an eye position zone with an area that is equal to or larger than said visual element;

wherein said at least one processor is configured to use said at least one eye position tracking sensor to determine when said at least one eye is on average gazing in a direction that is within the eye position zone of at least one of said visual elements for a time period exceeding a hovering time interval, signaling when said hovering time interval has been exceeded and selecting said at least one visual element;

wherein said at least one processor is also configured to register that a virtual key corresponding to said at least one of said selected visual elements has been activated by said user.

20. The device of claim 18, wherein said device additionally comprises a camera directed to observe an outside environment, and wherein said virtual display additionally displays augmented reality virtual information.

21. The device of claim 20, wherein said touchpad is a virtual touchpad that is displayed by said virtual display, and wherein said at least one camera and computer vision system is configured to recognize when a user is touching said virtual touchpad by imaging the position of a user hand, fingers, or other user controlled pointing device in a space observed by said camera.

22. The device of claim 18, wherein said device further comprises at least one transparent lens or other transparent device to allow visual information from an outside environment to be seen by at least one eye of said user.

23. The device of claim 18, wherein said device further comprises at least one camera disposed to view an outside environment, and connectivity to a computer vision system;

wherein said processor is further configured to acquire images from said at least one camera, and to use said computer vision system to recognize when said at least one camera is viewing a visual object, thereby producing an object recognition signal; and

wherein said processor is further configured to use said object recognition signal to control the display of at least some visual targets on said virtual display.