ENHANCED VIRTUAL TOUCHPAD

• US 20130283213A1
• Filed: 03/24/2013
• Published: 10/24/2013
• Est. Priority Date: 03/26/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Guendelman, Eran, Or, Ofir, Bychkov, Eyal, Brezner, Oren, Berenson, Adi
• Granted Patent: US 11,169,611 B2
• Time in Patent Office: Days
• US Class Current: 715/848
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A method, including receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface and segmenting the physical surface into one or more physical regions. A functionality is assigned to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, and a sequence of three-dimensional (3D) maps is received, the sequence of 3D maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions. The 3D maps are analyzed to detect a gesture performed by the user, and based on the gesture, an input is simulated for the tactile input device corresponding to the one of the physical regions.

Claims

1. A method, comprising:

receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface;

segmenting the physical surface into one or more physical regions;

assigning a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device;

receiving a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions;

analyzing the 3D maps to detect a gesture performed by the user; and

simulating, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.

2. The method according to claim 1, wherein the tactile input device is selected from a list comprising a keyboard, a mouse, and a touchpad.

3. The method according to claim 1, and comprising projecting an image on the physical surface in response to the gesture.

4. The method according to claim 3, wherein the tactile input device comprises a touchscreen.

5. The method according to claim 1, and comprising determining, based the sequence of 3D maps, a pressure applied by one or more fingers of the hand against the physical surface, and incorporating the pressure into the simulated input.

6. The method according to claim 1, and comprising identifying, based on the 2D image, a color of an object held by the hand and in contact with the physical surface, and incorporating the color into the simulated input.

7. The method according to claim 1, and comprising determining, based on the sequence of 3D maps, a position and a velocity of one or more fingers of the hand, and incorporating the position and the velocity into the simulated input.

8. An apparatus, comprising:

a sensing device configured to receive a two dimensional (2D) image containing at least a physical surface, and to receive a sequence of three dimensional (3D) maps containing at least a hand of a user, the hand positioned on the physical surface;

a display; and

a computer coupled to the sensing device and the display, and configured to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to analyze the 3D maps to detect a gesture performed by the user, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.

9. The apparatus according to claim 8, wherein the computer is configured to select the tactile input device from a list comprising a keyboard, a mouse, and a touchpad.

10. The apparatus according to claim 8, and comprising a projector coupled to the computer and configured to project an image on the physical surface in response to the gesture.

11. The apparatus according to claim 10, wherein the tactile input device comprises a touchscreen.

12. The apparatus according to claim 8, wherein the computer is configured to determine, based the sequence of 3D maps, a pressure applied by one or more fingers of the hand against the physical surface, and to incorporate the pressure into the simulated input.

13. The apparatus according to claim 8, wherein the computer is configured to identify, based on the 2D image, a color of an object held by the hand and in contact with the physical surface, and to incorporate the color into the simulated input.

14. The apparatus according to claim 8, wherein the computer is configured to determine, based on the sequence of 3D maps, a position and a velocity of one or more fingers of the hand, and to incorporate the position and the velocity into the simulated input.

15. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a two-dimensional image (2D) containing at least a physical surface, to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to receive a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions, to analyze the 3D maps to detect a gesture performed by the user, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.

16. A method, comprising:

receiving a sequence of three-dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface;

analyzing the 3D maps to detect a gesture performed by the user;

projecting, onto the physical surface, an animation in response to the gesture; and

incorporating the one or more physical objects into the animation.

17. The method according to claim 16, and comprising projecting a respective contour image encompassing each of the one or more physical objects, and incorporating the respective contour image into the animation.

18. An apparatus, comprising:

a sensing device configured to receive a sequence of three dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user, the hand positioned in proximity to the physical surface;

a projector; and

a computer coupled to the sensing device and the projector, and configured to analyze the 3D maps to detect a gesture performed by the user, to present, using the projector, an animation onto the physical surface in response to the gesture, and to incorporate the one or more physical objects into the animation.

19. The apparatus according to claim 18, wherein the computer is configured to present, using the projector, a respective contour image encompassing each of the one or more physical objects, and to incorporate the respective contour image into the animation.

20. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a sequence of three-dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface, to analyze the 3D maps to detect a gesture performed by the user, to project, onto the physical surface, an animation in response to the gesture, and to incorporate the one or more physical objects into the animation.

21. A method, comprising:

receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface;

segmenting the physical surface into one or more physical regions;

assigning a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device;

receiving a sequence of three-dimensional (3D) maps containing at least an object held by a hand of a user of the computer, the object positioned on one of the physical regions;

analyzing the 3D maps to detect a gesture performed using the object; and

simulating, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.

22. The method according to claim 21, and comprising identifying, based on the 2D image, a color of the object, and incorporating the color into the simulated input.

24. The apparatus according to claim 21, wherein the computer is configured to identify, based on the 2D image, a color of the object, and to incorporate the color into the simulated input.

23. An apparatus, comprising:

a sensing device configured to receive a two dimensional (2D) image containing at least a physical surface, and to receive a sequence of three dimensional (3D) maps containing at least an object held by a hand of a user, the object positioned on the physical surface;

a display; and

a computer coupled to the sensing device and the display, and configured to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to analyze the 3D maps to detect a gesture performed using the object, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.

25. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a two-dimensional image (2D) containing at least a physical surface, to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to receive a sequence of three-dimensional (3D) maps containing at least an object held by a hand of a user of the computer, the object positioned on one of the physical regions, to analyze the 3D maps to detect a gesture performed using the object, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions.