Multimodal interaction with near-to-eye display

• US 10,095,033 B2
• Filed: 07/27/2012
• Issued: 10/09/2018
• Est. Priority Date: 07/27/2012
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Wang, Kongqiao, Lantz, Vuokko, Li, Jiangwei, Xu, Lei, Huopaniemi, Jyri
• Primary Examiner(s): Yang, Nan-Ying
• Application Number:
  US14/412,357
• Publication Number:
  US 20150338651A1
• Time in Patent Office: 2,265 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34

Abstract

The invention relates to a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. The invention further relates to an apparatus and a computer program product configured to perform the method.

Claims

1. A method, comprising:

tracking a gaze of an eye of a user by an inside-looking camera of a near-to-eye display, wherein a position of the gaze determines a place for a cursor on the screen of the near-to-eye display, wherein the movement of the gaze is configured to move the cursor on the screen of the near-to-eye display and when the movement of the gaze is detected to be still or moving only slightly,

obtaining a first position of a hand of the user by a side down-looking camera of the near-to-eye display,

obtaining a second position of the hand of the user by the side down-looking camera of the near-to-eye display,

computing relative movement of the hand based on the first and second positions of the hand,

positioning the cursor on the basis of the relative movement of the hand,

in an instance at least one gesture of the hand is detected by the side down-looking camera, activating a manual control of the user interface by adjusting the position of the cursor on the screen of the near-to-eye display;

adjusting the position of the cursor on the screen of the near-to-eye display on a basis of further relative movement of the hand in an instance in which the cursor is outside an area of a menu option or a functional icon displayed on the screen; and

in an instance at least one gesture is no longer detected by the side down-looking camera, resuming a gaze control of the user interface.

2. The method of claim 1, wherein the further operation depends on the detected gesture.

3. The method of claim 1, wherein the detected gesture is a movement of a fist, shaking of a palm or movement of a palm from or to the second camera.

4. The method of claim 1, wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

5. The method of claim 1, wherein the further operation is:

selecting a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

6. The method of claim 1, wherein the further operation is:

zooming in or out the view of the screen of the near-to-eye display.

7. The method of claim 1, wherein the further operation is:

returning, removing or deleting.

8. The method of claim 1, further comprising:

observing blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and

selecting the menu option or the functional icon if blinking of the eye of the user is detected.

9. An apparatus comprising at least one processor, and at least one memory including a computer program code, said at least one memory and the computer program code being configured, with the processor, to cause the apparatus to perform at least the following:

track a gaze of one eye of a user by an inside-looking camera of a near-to-eye display, wherein a position of the gaze determines a place for a cursor on a screen of the near-to-eye display of the apparatus, wherein a movement of the gaze is configured to move the cursor on the screen of the near-to-eye display and when the movement of the gaze is detected to be still or moving only slightly,

obtain a first position of a hand of the user by a side down-looking camera of the near-to-eye display,

obtain a second position of the hand of the user by the side down-looking camera of the near-to-eye display,

compute relative movement of the hand based on the first and second positions of the hand,

position the cursor on the basis of the relative movement of the hand,

in an instance at least one gesture of the hand is detected by the side down-looking camera, activate a manual control of the user interface by adjusting the position of the cursor on the screen of the near-to-eye display;

adjust the position of the cursor on the screen of the near-to-eye display on a basis of further relative movement of the hand in an instance in which the cursor is outside an area of a menu option or a functional icon displayed on the screen; and

in an instance that the at least one gesture is no longer detected by the side down-looking camera, resume a gaze control of the user interface.

10. The apparatus of claim 9, wherein the further operation depends on the detected gesture.

11. The apparatus of claim 9, wherein the detected gesture may be the movement of a fist, the shaking of a palm and the movement of a palm from or to the second camera.

12. The apparatus of claim 9, wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

13. The apparatus of claim 9, wherein the further operation is:

select a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

14. The apparatus of claim 9, wherein the further operation is:

zoom in or out the view of the screen of the near-to-eye display.

15. The apparatus of claim 9, wherein the further operation is:

return, remove or delete.

16. The apparatus of claim 9, wherein the computer program code is further configured, with the processor, to cause the apparatus to perform:

observe blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and

select the menu option or functional icon of the cursor on the screen of the near-to-eye display if blinking of the eye of the user is detected.

17. The apparatus of claim 9, wherein said apparatus is a near-to-eye display device.

18. The apparatus of claim 17, wherein said near-to-eye display device is a terminal device for a mobile phone.