USER INTERFACE SYSTEM BASED ON POINTING DEVICE

• US 20110187643A1
• Filed: 03/31/2011
• Published: 08/04/2011
• Est. Priority Date: 11/20/2002
• Status: Active Grant

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: DEKKER, TIM, MEKENKAMP, GERHARDUS ENGBERTUS
• Granted Patent: US 8,971,629 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/61 Password, biometric
• CPC Class Codes: G08C 2201/71 Directional beams
• CPC Class Codes: G08C 2201/92 Universal remote control
• CPC Class Codes: G08C 2201/94 Smart cards

Abstract

The user interaction system comprises a portable pointing device (101) connected to a camera (102) and sending pictures to a digital signal processor (120), capable of recognizing an object (130) and a command given by the user (100) by moving the pointing device (101) in a specific way, and controlling an electrical apparatus (110) on the basis of this recognition.

Claims

1. User interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a region in space;

a camera taking a picture; and

a digital signal processor, capable of receiving and processing the picture, and capable of transmitting user interface information derived from the picture to the electrical apparatus, wherein the camera is connected to the pointing device so that in operation it images the region pointed to.

2. User interaction system as claimed in claim 1, wherein the user interface information includes apparatus control data for controlling operation of the electrical apparatus.

3. User interaction system as claimed in claim 1, wherein the digital signal processor comprises an object characterizing means for characterizing an object or part of the object present in the picture of the region imaged by the camera, by providing first object characterizing features to a comprised object identification means for identifying the object, and which object identification means is capable of outputting object identification data from which the user interface information is constructed.

4. User interaction system as claimed in claim 1, wherein the digital signal processor comprises:

motion trajectory estimation means for estimating a motion trajectory of the pointing device and outputting a first motion characterizing signature, a signature being a mathematical abstraction of the motion trajectory; and

signature identification means for identifying the first motion characterizing signature and outputting command identification data, which represents a user interaction command, corresponding with the first motion characterizing signature, from which command identification data the user interface information is constructed.

9. User interaction system as claimed in claim 4, wherein the first motion characterizing signature is derived on the basis of successive pictures imaged by the camera at respective instances of time.

10. Pointing device for use in a user interaction system as claimed in claim 1, characterized in that it comprises a camera and is capable of sending a picture to a digital signal processor.

11. Pointing device as claimed in claim 10, wherein the pointing device is capable of sending a picture to the digital signal processor, which is capable of sending user interface information to an electrical apparatus based on the picture.

12. Pointing device as claimed in claim 10 wherein the digital signal processor (120) is comprised in the pointing device.

13. Pointing device as claimed in claim 10, comprising motion sensing means for sensing a motion trajectory of the pointing device.

14. Pointing device as claimed in claim 10, comprising a characteristic projector for optically projecting a characteristic pattern towards a region pointed to.

15. Pointing device as claimed in claim 10, comprising a programmable user interface code generator and a wireless transmitter for transmitting the code to the electrical apparatus.

16. Pointing device as claimed in claim 10 comprising feedback means for feedback of user interface information.

17. Electrical apparatus for use in a user interaction system as claimed in claim 1, characterized in that interface means are comprised which allow the electrical apparatus to send information about supported commands to a pointing device as claimed in claim 1, based on an “

identify supported commands”

call of the pointing device to the electrical apparatus.

34. Electrical apparatus for use in a user interaction system as claimed in claim 18, characterized in that interface means are comprised which allow the electrical apparatus to send information about supported commands to a pointing device as claimed in claim 1, based on at least one call of the pointing device to the electrical apparatus.

5. (canceled)

6. (canceled)

7. (canceled)

8. (canceled)

18. User interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a region in space;

a camera) connected to the pointing device so that in operation it images the region pointed to for taking a picture;

motion sensing means for estimating the motion of the pointing device; and

a digital signal processor, capable of receiving and processing data of said picture, and capable of transmitting user interface information derived on the basis of said picture data to the electrical apparatus.

19. User interaction system as claimed in claim 18, comprising means for estimating the motion trajectory on the basis of the output of said motion-sensing means.

20. User interaction system as claimed in claim 18, wherein the motion of the pointing device is estimated on basis of successive pictures imaged by the camera at respective instances of time.

21. User interaction system as claimed in claim 18, wherein the motion sensing means is selected from the group consisting of a mass on a deformation sensor, a gyroscope and a differential GPS.

22. User interaction system as claimed in claim 18 wherein the transmitted user interface information includes at least one feature selected from the group consisting of motion speed, motion direction, and acceleration of the pointing device.

23. User interaction system as claimed in claim 19, wherein the transmitted user interface information includes at least one feature selected from the group consisting of motion trajectory of the pointing device and a characteristic signature derived from the motion trajectory of the pointing device.

24. User interaction system as claimed in claim 18, further comprising room localization beacons for emitting electromagnetic radiation, wherein the digital signal processor is arranged to recognize to which part of the room the pointing device is pointing.

25. User interaction system as claimed in claim 18, wherein the pointing device further comprises feedback means for providing the user with additional information.

26. User interaction system as claimed in claim 25, wherein said feedback means is selected from the group consisting of light, sound, a display and force feedback means.

27. Pointing device for use in a user interaction system as claimed in claim 18, comprising a camera and being capable of sending a picture to a digital signal processor.

28. Pointing device as claimed in claim 27, wherein the pointing device is capable of sending a picture to the digital signal processor, which is capable of sending user interface information to an electrical apparatus based on the picture.

29. Pointing device as claimed in claim 27 wherein the digital signal processor is comprised in the pointing device.

30. Pointing device as claimed in claim 27, comprising motion sensing means for sensing a motion trajectory of the pointing device.

31. Pointing device as claimed in claim 27, comprising a characteristic projector for optically projecting a characteristic pattern towards a region pointed to.

32. Pointing device as claimed in claim 27, comprising a programmable user interface code generator and a wireless transmitter for transmitting the code to the electrical apparatus.

33. Pointing device as claimed in claim 27 comprising feedback means for feedback of user interface information.

35. User interaction system, comprising:

a first electrical apparatus;

a portable pointing device operable by a user for pointing to a region in space;

a camera) connected to the pointing device so that in operation it images the region pointed to for taking a picture;

a digital signal processor, capable of receiving and processing data of said picture, and capable of transmitting user interface information derived on the basis of said picture data to the first electrical apparatus; and

an object displaying a characteristic pattern to facilitate recognition of said object by the digital signal processor.

36. User interaction system as claimed in claim 35, wherein the object displaying a characteristic pattern is a second electrical apparatus.

37. User interaction system as claimed in claim 35, wherein the second electrical apparatus comprises a display.

38. User interaction system as claimed in claim 35 in which the second electrical apparatus is in fact the first electrical apparatus.

39. Pointing device for use in a user interaction system as claimed in claim 35, comprising a camera and being capable of sending a picture to a digital signal processor.

40. Pointing device as claimed in claim 39, wherein the pointing device is capable of sending a picture to the digital signal processor, which is capable of sending user interface information to an electrical apparatus based on the picture.

41. Pointing device as claimed in claim 39 wherein the digital signal processor is comprised in the pointing device.

42. Pointing device as claimed in claim 39, comprising motion sensing means for sensing a motion trajectory of the pointing device.

43. Pointing device as claimed in claim 39, comprising a characteristic projector for optically projecting a characteristic pattern towards a region pointed to.

44. Pointing device as claimed in claim 39, comprising a programmable user interface code generator and a wireless transmitter for transmitting the code to the electrical apparatus.

45. Pointing device as claimed in claim 39 comprising feedback means for feedback of user interface information.

46. Electrical apparatus for use in a user interaction system as claimed in claim 35, wherein interface means allow the electrical apparatus to send information about supported commands to a pointing device, based on at least one identify supported commands call of the pointing device to the electrical apparatus.

47. User interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a region in space;

a camera connected to the pointing device so that in operation it images the region pointed to for taking a picture;

a digital signal processor, capable of receiving and processing data of said picture, and capable of transmitting user interface information derived on the basis of said picture data to the electrical apparatus; and

wherein said user interface information includes a specification of the electrical apparatus, or of a part of the electrical apparatus intended to be used by the user.

48. User interaction system as claimed in claim 47, wherein the electrical apparatus, or the intended part of the electrical apparatus is identified by determining the position of the electrical apparatus in the picture of the region pointed to.

49. User interaction system as claimed in claim 48, wherein the electrical apparatus, or the intended part of the electrical apparatus corresponds to a fixed position in the picture.

50. User interaction system as claimed in claim 47, further comprising means for feedback generation on basis of said transmitted user interface information.

51. User interaction system as claimed in claim 50, wherein the means for feedback generation include the generation of visual/auditory signals in proximity to the electrical apparatus, or to the intended part of the electrical apparatus.

52. Pointing device for use in a user interaction system as claimed in claim 47, comprising a camera and being capable of sending a picture to a digital signal processor.

53. Pointing device as claimed in claim 52, wherein the pointing device is capable of sending a picture to the digital signal processor, which is capable of sending user interface information to an electrical apparatus based on the picture.

54. Pointing device as claimed in claim 52 wherein the digital signal processor is operatively associated with the pointing device.

55. Pointing device as claimed in claim 52, comprising motion sensing means for sensing a motion trajectory of the pointing device.

56. Pointing device as claimed in claim 52, comprising a characteristic projector for optically projecting a characteristic pattern towards a region pointed to.

57. Pointing device as claimed in claim 52, comprising a programmable user interface code generator and a wireless transmitter for transmitting the code to the electrical apparatus.

58. Pointing device as claimed in claim 52 comprising feedback means for feedback of user interface information.

59. Electrical apparatus for use in a user interaction system as claimed in claim 47, comprising means allowing the electrical apparatus to send information about supported commands to a pointing device, based on an “

identify supported commands”

call of the pointing device to the electrical apparatus.

60. User interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a region in space;

a camera taking a picture; and

a digital signal processor, capable of receiving and processing the picture, and capable of transmitting user interface information derived from the picture to the electrical apparatus, wherein the camera is connected to the pointing device so that in operation it images the region pointed to characterized in that the system further comprises at least one localization beacon that can emit electromagnetic radiation, which can be captured by the system and for use to the digital signal processor in order to recognize to where the pointing device is pointing by using information derived from the captured electromagnetic radiation.

61. The user interaction system as in claim 60, wherein the digital signal processor (120) is further arranged to recognize the direction of the pointer relative to the localization beacon.

62. The user interaction system as in claim 60, further comprising motion sensing means for enabling estimating a motion or a motion trajectory of the pointing device.

63. The user interaction system as in claim 60, wherein the motion or the motion trajectory of the pointing device is estimated on basis of successive pictures imaged by the camera at respective instances of time.

64. User interaction system as claimed in claim 61 wherein the transmitted user interface information includes at least one feature selected from the group consisting of motion speed, and motion direction of the pointing device.

65. User interaction system as claimed in claim 62, wherein the transmitted user interface information includes at least one feature selected from the group consisting of motion trajectory of the pointing device and a characteristic signature derived from the motion trajectory of the pointing device.

66. User interaction system as claimed in one of claim 60, wherein the pointing device (101) further comprises feedback means for providing the user with additional information.

67. User interaction system as claimed in claim 66, wherein said feedback means is selected from the group consisting of light, sound, a display and force feedback means.

68. User interaction system as in one of claim 60, wherein the user interface information comprises apparatus control data for controlling operation of the electrical apparatus.

69. Pointing device for use in a user interaction system as in one of claim 60, characterized in that it comprises the digital signal processor and a camera and wherein the pointing device is capable of sending a picture to the digital signal processor.

70. Pointing device as in claim 69, comprising a characteristic projector for optically projecting a characteristic pattern towards a region pointed to.

71. Pointing device as in claim 60, comprising a programmable user interface code generator and a wireless transmitter for transmitting the code to the electrical apparatus.

72. Electrical apparatus for use in a user interaction system as in claim 60, characterized in that interface means are comprised which allow the electrical apparatus to send information about supported commands to a pointing device, based on an “

identify supported commands”

call of the pointing device to the electrical apparatus.