MULTIMODAL INTERATION WITH NEAR-TO-EYE DISPLAY

• US 20150338651A1
• Filed: 07/27/2012
• Published: 11/26/2015
• Est. Priority Date: 07/27/2012
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: WANG, Kongqiao, LI, Jiangwei, XU, Lei, LANTZ, Vuokko, HUOPANIEMI, Jyri
• Granted Patent: US 10,095,033 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34

Abstract

The invention relates to a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. The invention further relates to an apparatus and a computer program product configured to perform the method.

Claims

1-34. -34. (canceled)

35. A method, comprising:

tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display, wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly,

observing a gesture of one hand of the user by a second camera, and

executing a further operation if at least one gesture of the hand is detected.

36. A method of claim 35, wherein the further operation depends on the detected gesture.

37. A method of claim 35, wherein the detected gesture is a movement of a fist, shaking of a palm or movement of a palm from or to the second camera.

38. A method of claim 35, wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

39. A method of claim 35, wherein the further operation is:

adjusting the position of the cursor on the screen of the near-to-eye display on the basis of the relative movement of the hand if the cursor is outside the area of a menu option or a functional icon displayed on the screen.

40. A method of claim 35, wherein the further operation is:

selecting a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

41. A method of claim 35, wherein the further operation is:

zooming in or out the view of the screen of the near-to-eye display.

42. A method of claim 35, wherein the further operation is:

returning, removing or deleting.

43. A method of claim 35, further comprising:

observing blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and

selecting the menu option or the functional icon if blinking of the eye of the user is detected.

44. An apparatus comprising at least one processor, at least one memory including a computer program code for one or more program units, said at least one memory and the computer program code being configured, with the processor, to cause the apparatus to perform at least the following:

track the gaze of one eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on a screen of a near-to-eye display of the apparatus, wherein a movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly,

observe a gesture of one hand of the user by a second camera, and

execute a further operation if at least one gesture of the hand is detected.

45. An apparatus of claim 44, wherein the further operation depends on the detected gesture.

46. An apparatus of claim 44, wherein the detected gesture may be the movement of a fist, the shaking of a palm and the movement of a palm from or to the second camera.

47. An apparatus of claim 44, wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

48. An apparatus of claim 44, wherein the further operation is:

adjust the position of the cursor on the screen of the near-to-eye display on the basis of the relative movement of the hand if the cursor is outside the area of a menu option or a functional icon displayed on the screen.

49. An apparatus of claim 44, wherein the further operation is:

select a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

50. An apparatus of claim 44, wherein the further operation is:

zoom in or out the view of the screen of the near-to-eye display.

51. An apparatus of claim 44, wherein the further operation is:

return, remove or delete.

52. An apparatus of claim 44, wherein the computer program code is further configured, with the processor, to cause the apparatus to perform:

observe blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and

select the menu option or functional icon of the cursor on the screen of the near-to-eye display if blinking of the eye of the user is detected.

53. The apparatus of claim 44, wherein said apparatus is a near-to-eye display device.

54. The apparatus of claim 53, wherein said near-to-eye display device is a terminal device for a mobile phone.