Visual feedback by identifying anatomical features of a hand

• US 9,317,130 B2
• Filed: 06/18/2012
• Issued: 04/19/2016
• Est. Priority Date: 06/16/2011
• Status: Active Grant

• Current Assignee: Krepec, Rafal Jan
• Original Assignee: Krepec, Rafal Jan
• Inventors: Krepec, Rafal Jan
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/404,468
• Publication Number:
  US 20150153832A1
• Time in Patent Office: 1,401 Days
• Field of Search: 345156-183, 178 1801- 1809, 178 2001- 2004
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041

Abstract

A method (62) for providing visual feedback to a user having a hand (14) and operating a graphical user interface (28) on a display (24) using an input device (10). The input device (10) includes an image sensor (46) and an input element (40) defining an input surface (42), the input element (40) being at least partially transparent. The method (62) includes: with the hand (14) substantially adjacent to the input element (40), acquiring a hand image of the hand (14) through the input element (40) using the image sensor (46); identifying predetermined anatomical features (13) in the hand image and associate with each a corresponding feature location indicative of a position of the predetermined anatomical features (13) relative to the input surface (40); and displaying cursors (34, 61) in a window (30) of the graphical user interface (28) at positions relative to the window (30) that correspond to the feature locations relative to the input surface (40).

Claims

1. A system usable by an intended user having a hand to operate a graphical user interface, said hand defining predetermined anatomical features, said graphical user interface including a window, said system comprising:

a display;

an input device, said input device including;

an input element defining an input surface and an opposed undersurface, said input element being at least partially transparent; and

an image sensor configured located and operative for capturing a hand image of said hand through said input element when said hand is located substantially adjacent to said input surface and either entirely spaced apart therefrom or contacting said input surface;

an image processor coupled to said image sensor and operative for receiving said hand image from said image sensor and for processing said hand image to identify said predetermined anatomical features of said hand in said hand image and associate with each of said predetermined anatomical features a corresponding feature location, said feature locations being indicative of positions of said predetermined anatomical features relative to said input surface; and

a display interface coupled to said image processor and to said display and operative for displaying on said display said window and displaying cursors in said window, said cursors being each associated with a corresponding ones of said predetermined anatomical features, said cursors being positioned so that the position of said cursors relative to said window corresponds to said feature locations relative to said input surface.

2. The system as defined in claim 1, wherein said image sensor includes an array of imaging elements provided substantially adjacent to said undersurface, said undersurface being provided between said image sensor and said input surface.

3. The system as defined in claim 2, wherein said hand image includes an array of pixels, each of said pixels being acquired by a respective one of said imaging elements.

4. The system as defined in claim 2, wherein said array of imaging elements and said input surface have substantially similar dimensions.

5. The system as defined in claim 2, wherein said imaging elements are near field semiconductor light detection elements.

6. The system as defined in claim 5, wherein said imaging elements are complementary metal oxide semiconductor (CMOS) imaging elements.

7. The system as defined in claim 1, wherein

said image processor is further operative for processing said hand image to derive a processed hand image therefrom, said processed hand image preserving a shape and orientation of said hand image;

said display interface is operative for overlaying on said display said processed hand image in a manner such that the position and orientation of said processed hand image relative to said window corresponds to the position and orientation of said hand relative to the input surface.

8. The system as defined in claim 1, wherein said display and said input surface have different spatial orientations.

9. The system as defined in claim 8, wherein said display and said input surface are substantially perpendicular to each other.

10. The system as defined in claim 9, wherein said display is substantially vertical and said input surface is substantially horizontal.

11. The system as defined in claim 1, wherein said input surface is contact sensitive and said input element is operative for providing contact coordinates indicative of a contact location at which said hand contacts said input surface, said display interface being operative for receiving said contact location and displaying an alternative cursor in said window at an alternative cursor location, said alternative cursor being positioned so that said alternative cursor location relative to said window corresponds to said contact location relative to said input surface.

12. The system as defined in claim 1 wherein said input element is contact sensitive and said hand contacts said input surface at a contact location, said input element being operative for providing contact coordinates indicative of said contact location to said graphical user interface.

13. The system as defined in claim 12, wherein said input element detects contact by a detection method selected from the group consisting of a capacitive contact detection method, a resistive contact detection method, a detection method using surface acoustic waves, an infrared contact detection method, an optical imaging contact detection method, a dispersive signal technology detection method and an acoustic pulse recognition method.

14. The system as defined in claim 1, further comprising a stylus defining a stylus end point, said stylus being graspable by said hand, said input surface being contact sensitive and said input element being operative for providing contact coordinates indicative of a contact location at which said stylus end point contacts said input surface, said display interface being operative for receiving said contact location and displaying an alternative cursor in said window at an alternative cursor location, said alternative cursor being positioned so that said alternative cursor location relative to said window corresponds to said contact location relative to said input surface.

15. The system method as defined in claim 1, further comprising a stylus defining a stylus end point, said stylus being graspable by said hand, said input element being contact sensitive for sensing a contact location at which said stylus end point contacts said input surface, said input element being further operative for providing contact coordinates indicative of said contact location to said graphical user interface.

16. The system as defined in claim 1, wherein said input device further comprises an input device display substantially in register with said input surface, said display interface being operative for displaying said window on said input device display.

17. The system as defined in claim 1, wherein said input element is entirely transparent in register with said image sensor.

18. The system as defined in claim 1, wherein said image processor is operative for classifying said predetermined anatomical features as either contacting said input element or being spaced apart therefrom.

19. The system as defined in claim 18, wherein said graphical user interface includes an object, said image processor being operative for highlighting said object if at least one of said cursors is substantially adjacent thereto, said at least one of said cursors being associated with one of said predetermined anatomical features that is spaced apart from said input surface.

20. An input device usable with a computing device by an intended user having a hand, said input device comprising:

an input element defining an input surface and an opposed undersurface, said input element being at least partially transparent;

an image sensor for capturing through said input element a hand image of said hand when said hand is located substantially adjacent to said input surface and either entirely spaced apart therefrom or contacting said input surface;

an input device interface couplable to said computing device and operative for transmitting to said computing device image information obtained from said hand image;

wherein said hand defines at least two predetermined anatomical features, said image processor being operative for receiving said hand image from said image sensor and for processing said hand image to identify said at least two predetermined anatomical features of said hand in said hand image and associate with each of said at least two predetermined anatomical features a corresponding feature location, said feature locations being indicative of a position of said at least two predetermined anatomical features relative to said input surface; and

said image information includes said feature locations.

21. The input device as defined in claim 20, wherein said image information includes an unprocessed version of said hand image.

22. The input device as defined in claim 20, further comprising an image processor coupled to said image sensor and to said input device interface and operative for

receiving said hand image from said image sensor;

processing said hand image to compute said image information; and

releasing said image information to said input device interface.

23. The input device as defined in claim 22, wherein said image information includes a contour of said hand in said hand image.

24. The input device as defined in claim 22, wherein

said hand defines at least one predetermined anatomical feature, said image processor being operative for receiving said hand image from said image sensor and for processing said hand image to identify said at least one predetermined anatomical feature of said hand in said hand image and associate with said at least one predetermined anatomical features a feature location, said feature location being indicative of a position of said at least one predetermined anatomical feature relative to said input surface;

and

said image information includes said feature location.

25. The input device as defined in claim 24, wherein said image processor is operative for classifying said predetermined anatomical features as either contacting said input element or being spaced apart therefrom, said image information including an indication of whether each of said predetermined anatomical feature contacts or is spaced apart from said input element.

26. The input device as defined in claim 20, wherein said image sensor includes an array of imaging elements provided substantially adjacent to said undersurface, said undersurface being provided between said image sensor and said input surface.

27. The input device as defined in claim 26, wherein said hand image includes an array of pixels, each of said pixels being acquired by a respective one of said imaging elements.

28. The input device as defined in claim 27, wherein said array of imaging elements and said input surface have substantially similar dimensions.

29. The input device as defined in claim 26, wherein said imaging elements are near field semiconductor light detection elements.

30. The input device as defined in claim 29, wherein said imaging elements are complementary metal oxide semiconductor (CMOS) imaging elements.

31. The input device as defined in claim 20, wherein said input surface is contact sensitive and said input element is operative for providing to said input device interface contact coordinates indicative of a contact location at which said hand contacts said input surface, said input device interface being operative for receiving said contact location and transmitting to said computing device said contact location.

32. The input device as defined in claim 31, wherein said contact sensitive surface detects contact by a detection method selected from the group consisting of a capacitive contact detection method, a resistive contact detection method, a detection method using surface acoustic waves, an infrared contact detection method, an optical imaging contact detection method, a dispersive signal technology detection method and an acoustic pulse recognition method.

33. The input device as defined in claim 20, wherein said input device is usable with a stylus held by said hand, said input surface being contact sensitive and said input element being operative for providing to said input device interface contact coordinates indicative of a contact location at which said stylus end point contacts said input surface, said input device interface being operative for receiving said contact location and transmitting to said computing device said contact location.

34. The input device as defined in claim 20, wherein said input device is usable with a stylus held by said hand, said input surface being contact sensitive and said input element being operative for providing to said input device interface contact coordinates indicative of contact locations at which said hand and said stylus end point contact said input surface, said input device interface being operative for receiving said contact locations and transmitting to said computing device said contact locations.

35. The input device as defined in claim 20, further comprising an input device display substantially in register with said input surface, said input device interface being operative for receiving an image from said computing device and displaying said image on said input device display.

36. The input device as defined in claim 20, wherein said input element is entirely transparent in register with said image sensor.