User interface

• US 6,771,294 B1
• Filed: 12/29/1999
• Issued: 08/03/2004
• Est. Priority Date: 12/29/1999
• Status: Expired due to Term

• Current Assignee: Augumenta Oy
• Original Assignee: Peter Antoniac, Petri Pulli
• Inventors: Pulli, Petri, Antoniac, Peter
• Primary Examiner(s): Kincaid, Kristine
• Assistant Examiner(s): TRAN, MYLINH T
• Application Number:
  US09/473,709
• Time in Patent Office: 1,679 Days
• Field of Search: 345/863, 345/835, 345/764, 345/810, 345/173, 345/155, 345/156
• US Class Current: 715/863
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The invention relates to a method of using a portable mobile communication device such as mediaphone. In the method, an image of a hand of the user is made and the natural partition of the hand is recognized. An augmented reality image of a user interface of the device is made by laying an image of a desired user interface comprising input segments onto the image of the user'"'"'s hand in such a way that the segments of the user interface are separated from each other by the natural partition of the hand. The augmented reality image of the user interface is seen by the user, who selects a desirable segment by touching a partition on the hand. The selection on the hand is recognized and a function related to the selected segment is performed by the device. MARISIL is defined and used to operate this class of devices.

Claims

1. A method of using a portable mobile communication device, the method comprising:

forming an image of a physical hand of a user and recognizing a natural partition of the user'"'"'s physical hand;

forming an augmented reality image by overlaying an image of a desired user interface on an image of the user'"'"'s physical hand formed by rays of light originating from the user'"'"'s physical hand and entering the user'"'"'s eyes;

selecting a desired segment by touching a suitable partition on the user'"'"'s physical hand, the segment being recognized as a natural partition of the user'"'"'s physical hand; and

performing a function related to the segment in the device.

2. A method according to claim 1, wherein the desired segment is selected by the user.

3. A method according to claim 1, wherein the image of the desired user interface, overlaid on the image of the user'"'"'s physical hand recognized by the user with his eyes, is formed by means of digital signal processing from data provided by virtual reality gloves.

4. A method according to claim 1, wherein the user interface is an alphanumerical user interface.

5. A method according to claim 1, wherein the user interface is an interface based on QFD symbols.

6. A method according to claim 1, wherein the image of the user'"'"'s physical hand is formed as a video image.

7. A method according to claim 1, wherein the image of a desired user interface consists of computer graphics.

8. A method according to claim 1, wherein the user interface comprises input segments and is situated on the image of the user'"'"'s physical hand in such a way that the segments of the user interface are separated from each other by the natural partition of the user'"'"'s physical hand.

9. A method according to claim 1, wherein the device is operated by means of MARISIL sign language.

10. A method of using a portable mobile communication device, the method comprising:

forming an image of a physical hand of a user and recognizing the a natural partition of the user'"'"'s physical hand;

forming an augmented reality image of a user interface of the device by overlaying an image of a desired user interface on the image of the user'"'"'s physical hand;

showing a user the augmented reality image of the user interface;

selecting a desired segment by touching a suitable partition on the user'"'"'s physical hand, the segment being recognized as a natural partition of the user'"'"'s physical hand;

recognizing the selection on the user'"'"'s physical hand; and

performing a function related to the segment in the device.

11. A method according to claim 10, wherein the desired segment is selected by the user.

12. A method according to claim 10, wherein the image of the desired user interface, overlaid on the image of the user'"'"'s physical hand, is formed by means of digital signal processing from data provided by virtual reality gloves.

13. A method according to claim 10, wherein the user interface is an alphanumerical user interface.

14. A method according to claim 10, wherein the user interface is an interface based on QFD symbols.

15. A method according to claim 10, wherein the image of the user'"'"'s physical hand is formed as a video image.

16. A method according to claim 10, wherein the image of a desired user interface consists of computer graphics.

17. A method according to claim 10, wherein the user interface comprises input segments and is situated on the image of the user'"'"'s physical hand in such a way that the segments of the user interface are separated from each other by the natural partition of the user'"'"'s physical hand.

18. A method according to claim 10, wherein the device is operated by means of MARISIL sign language.

19. A portable mobile communication device comprising a video camera unit, a display unit, a transceiver unit and a digital signal processing unit that is arranged to control the operation of the portable mobile communication device, wherein:

the video camera is arranged to form a video image of a user'"'"'s physical hand and to feed the image into the digital signal processing unit, which is arranged to recognize a natural partition of the user'"'"'s physical hand in the video image;

the digital signal processing unit is arranged to feed an image of a user interface into the display and to form an augmented reality image of the user interface of the device by overlaying an image of a desired user interface on an image of the user'"'"'s physical hand formed by rays of light originating from the user'"'"'s physical hand, passing through the display, and entering the user'"'"'s eyes;

a desired segment, recognized as a natural partition of the user'"'"'s physical hand, is selected by the user by touching a suitable partition on the user'"'"'s physical hand;

the digital signal processing unit is arranged to recognize the selection on the user'"'"'s physical hand; and

the device is arranged to perform a function related to the segment.

20. A device according to claim 19, wherein the user selects a desired segment.

21. A device according to claim 19, wherein the user interface is an alphanumerical user interface.

22. A device according to claim 19, wherein the user interface is an interface based on QFD symbols.

23. A device according to claim 19, wherein the image of a desired user interface consists of computer graphics.

24. A device according to claim 19, wherein the user interface comprises input segments and is situated on the image of the user'"'"'s physical hand in such a way that the segments of the user interface are separated from each other by the natural partition of the user'"'"'s physical hand.

25. A device according to claim 19, wherein the device is arranged to operate by means of MARISIL sign language.

26. A portable mobile communication device comprising a video camera unit, a display unit, a transceiver unit and a digital signal processing unit that is arranged to control the operation of the portable mobile communication device, wherein:

the video camera is arranged to form a video image of a user'"'"'s physical hand and to feed the image into the digital signal processing unit, which is arranged to recognize a natural partition of the user'"'"'s physical hand in the video image;

the digital signal processing unit is arranged to form an augmented reality image of a user interface of the device by overlaying an image of a desired user interface on the image of the user'"'"'s physical hand;

the digital signal processing unit is arranged to feed the augmented reality image of the user interface into the display, which is arranged to show the image to the user;

a desired segment, recognized as a natural partition of the user'"'"'s physical hand, is selected by the user by touching a suitable partition on the user'"'"'s physical hand;

the digital signal processing unit is arranged to recognize the selection on the user'"'"'s physical hand; and

the device is arranged to perform a function related to the segment.

27. A device according to claim 26, wherein the user selects a desired segment.

28. A device according to claim 26, wherein the user interface is an alphanumerical user interface.

29. A device according to claim 26, wherein the user interface is an interface based on QFD symbols.

30. A device according to claim 26, wherein the image of a desired user interface consists of computer graphics.

31. A device according to claim 26, wherein the user interface comprises input segments and is situated on the image of the user'"'"'s physical hand in such a way that the segments of the user interface are separated from each other by the natural partition of the user'"'"'s physical hand.

32. A device according to claim 26, wherein the device is arranged to operate by means of MARISIL sign language.

33. A portable mobile communication device comprising a virtual reality gloves unit, a position tracking unit, a display unit, a transceiver unit and a digital signal processing unit that is arranged to control the operation of the portable mobile communication device, wherein:

the virtual reality gloves are arranged to feed information on a user'"'"'s physical hand into the digital signal processing unit, which is arranged to form an image of the user'"'"'s physical hand and to recognize a natural partition of the user'"'"'s physical hand;

the digital signal processing unit is arranged to form an augmented reality image of a user interface of the device by overlaying an image of a desired user interface on the image of the user'"'"'s physical hand;

the digital signal processing unit is arranged to feed the augmented reality image of the user interface into the display, which is arranged to show the image to the user;

a desired segment, recognized as a natural partition of the user'"'"'s physical hand, is selected by the user by touching a suitable partition on the user'"'"'s physical hand;

the digital signal processing unit is arranged to recognize the selection on the user'"'"'s physical hand; and

the device is arranged to perform a function related to the segment.

34. A device according to claim 33, wherein the user selects a desired segment.

35. A device according to claim 33, wherein the user interface is an alphanumerical user interface.

36. A device according to claim 33, wherein the user interface is an interface based on QFD symbols.

37. A device according to claim 33, wherein the image of a desired user interface consists of computer graphics.

38. A device according to claim 33, wherein the user interface comprises input segments and is situated on the image of the user'"'"'s physical hand in such a way that the segments of the user interface are separated from each other by the natural partition of the user'"'"'s physical hand.

39. A device according to claim 33, wherein the device is arranged to operate by means of MARISIL sign language.