Camera assisted pen tablet

• US 7,969,409 B2
• Filed: 05/19/2005
• Issued: 06/28/2011
• Est. Priority Date: 02/18/2004
• Status: Expired due to Fees

• Current Assignee: Intellectual Ventures Assets 21 LLC (Intellectual Ventures LLC)
• Original Assignee: Rafal Jan Krepec
• Inventors: Krepec, Rafal Jan
• Primary Examiner(s): Nguyen; Kevin M
• Assistant Examiner(s): Lee, Jr.; Kenneth B
• Application Number:
  US11/133,141
• Publication Number:
  US 20070024590A1
• Time in Patent Office: 2,231 Days
• Field of Search: 345156-172, 345418-475, 705 1- 45, 725 37- 61, 700/715
• US Class Current: 345/156
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/08 Cursor circuits

Abstract

An x-y user input device, such as a pen tablet or digitizing tablet, used to control a cursor on a display of a data processing system, comprises an imaging device or camera and an image processor for displaying a captured image of a user'"'"'s hand operating the input device within a window of the display. The input device is generally operated using a pointing device that controls the cursor position on the display while being synchronously imaged and displayed with the user'"'"'s hand on the display. The position of the captured image of the user'"'"'s hand and pointing device in the window relative to the cursor corresponds to a position of the hand relative to the pointing device, thereby providing a visualization effect that increases user comfort and proficiency with the input device.

Claims

1. A method for providing visual feedback to a user operating a pointing device on a tablet for controlling a cursor within a window of a display, the user using a physical hand to operate the pointing device, the tablet defining a tablet region corresponding to the window, the tablet region comprising dimensions proportionally the same as the window, said method using an imaging device, said method comprising:

acquiring with the tablet a pointing device position of the pointing device relative to the tablet region;

substantially simultaneously to acquiring said pointing device position, capturing with the imaging device a hand image of the physical hand of the user operating the pointing device;

positioning the cursor in the window using said pointing device position, the cursor being positioned so that the position of the cursor relative to the window corresponds to the pointing device position relative to the tablet region; and

substantially simultaneously to positioning the cursor in the window, overlaying on the display said hand image in a manner such that the position and orientation of said hand image relative to the cursor and to the window corresponds to the position and orientation of the hand relative to the pointing device position and to the tablet region;

whereby visual feedback regarding the position, orientation and shape of the physical hand of the user operating the pointing device is provided directly on the display by overlaying on the display the hand image acquired of the physical hand of the user operating the pointing device.

2. The method as in claim 1, wherein the window comprises dimensions substantially the same as the display.

3. The method as in claim 1, wherein the window comprises dimensions less than the display.

4. The method as in claim 1, wherein the window comprises at least one sub-window.

5. The method as in claim 4, wherein said hand image is displayed only when the cursor is within at least one of said at least one sub-window.

6. The method as in claim 1, wherein said step of displaying said hand image comprises a step of mixing said captured image with a second image in the window.

7. The method as in claim 6, wherein said second image comprises a graphical user interface.

8. The method as in claim 6, wherein said step of mixing said hand image with a second image comprises selectively adjusting a translucence of said hand image such that when said captured image overlays a fraction of said second image, said fraction can still be seen through said hand image.

9. The method as in claim 6, wherein said step of mixing said hand image with a second image comprises selectively adjusting said position of said hand image to selectively align a point of said hand image with the cursor.

43. A method as defined in claim 1, wherein acquiring said hand image includes

capturing a captured image of the tablet and of the physical hand positioned over the tablet;

and

extracting from the captured image the portion of the captured image corresponding to the physical hand while preserving shape, position and orientation of the physical hand of the user.

44. A method as defined in claim 1, further comprising:

providing the tablet and the display with the display and the tablet having spatial orientations that differ from each other.

10. A user input device for providing visual feedback to a user operating a pointing device for controlling a cursor within a window of a display of a data processing system, the user using a physical hand to operate the pointing device, said user input device comprising:

a tablet, said tablet defining a tablet region corresponding to the window, said tablet region comprising dimensions proportionally the same as the window, said tablet including an active surface, said active surface providing pointing device position information indicative of the pointing device position relative to said tablet region an imaging device for capturing a hand image of the physical hand of the user operating the pointing device substantially simultaneously to acquiring said pointing device position; and

an image processor for positioning the cursor in the window using said pointing device position, the cursor being positioned so that the position of the cursor relative to the window corresponds to the pointing device position relative to the tablet region and, substantially simultaneously to positioning the cursor in the window, overlaying on the display said hand image in a manner such that the position and orientation of said hand image relative to the cursor and to the window corresponds to the position and orientation of the hand relative to the pointing device position and to the tablet region whereby visual feedback regarding the position, orientation and shape of the physical hand of the user operating the pointing device is provided directly on the display by overlaying on the display the hand image acquired of the physical hand of the user operating the pointing device.

11. The user input device as in claim 10, wherein the window comprises dimensions substantially the same as the display.

12. The user input device as in claim 10, wherein the window comprises dimensions less than the display.

13. The user input device as in claim 10, wherein the window comprises at least one sub-window.

14. The user input device as in claim 13, wherein said hand image is displayed only when the cursor is within at least one of said at least one sub-window.

15. The user input device as in claim 10, wherein said image processor comprises an image mixer for combining said hand image with a second image.

16. The user input device as in claim 15, wherein said second image comprises a graphical user interface.

17. The user input device as in claim 15, wherein said image mixer comprises mixing software.

18. The user input device as in claim 17, wherein said mixing software comprises at least one adjustable image mixing parameter.

19. The user input device as in claim 18, wherein said at least one adjustable image mixing parameter comprises a translucence parameter for selectively adjusting a translucence of said hand image.

20. The user input device as in claim 18, wherein said at least one adjustable image mixing parameter comprises an alignment parameter for selectively adjusting said position of said hand image on the display.

21. The user input device as in claim 17, wherein said software is implemented in the data processing system.

22. The user input device as in claim 15, wherein said image mixer comprises mixing hardware.

23. The user input device as in claim 22, wherein said mixing hardware comprises at least one adjustable image mixing switch.

24. The user input device as in claim 10, wherein said at least one adjustable image mixing switch comprises a translucence switch for selectively adjusting a translucence of said hand image.

25. The user input device as in claim 10, wherein said at least one adjustable image mixing switch comprises an alignment switch for selectively adjusting said position of said captured image on the display.

26. The user input device as in claim 10, wherein said hand image is translucent.

27. The user input device as in claim 10, wherein said hand image is opaque.

28. The user input device as in claim 10, wherein said imaging device is an optical device.

29. The user input device as in claim 28, wherein said optical device is a camera.

30. The user input device as in claim 10 wherein acquiring said hand image includes capturing a captured image of said tablet and of the physical hand positioned over said tablet;

and extracting from the captured image the portion of the captured image corresponding to the physical hand while preserving shape, position and orientation of the physical hand of the user.

31. The user input device as in claim 10, wherein said display and said tablet have spatial orientations that differ from each other.

32. The user input device as in claim 10, wherein said imaging device is a heat sensing device.

33. The user input device as in claim 10, wherein said imaging device is a sonar device.

34. The user input device as in claim 10, wherein said imaging device is a loser scanning device.

35. The user input device as in claim 10, wherein said imaging device is an IR imaging device.

36. The user input device as in claim 10, wherein said imaging device is positioned above said tablet region.

37. The user input device as in claim 36, wherein a position of said imaging device is substantially centered above said tablet region.

38. The user input device as in claim 36, wherein an optical axis of said imaging device is substantially normal to said tablet.

39. The user input device as in claim 10, wherein said pointing device is a pen-like object.

40. The user input device as in claim 10, wherein said pointing device is a stylus.

41. The user input device as in claim 10, wherein said pointing device is a mouse.

42. The user input device as in claim 10, wherein said pointing device is a user finger.