Virtual touchpads for wearable and portable devices

• US 10,444,908 B2
• Filed: 12/30/2017
• Issued: 10/15/2019
• Est. Priority Date: 12/31/2016
• Status: Active Grant

• Current Assignee: Innoventions Incorporated
• Original Assignee: Innoventions Incorporated
• Inventors: Feinstein, David Y.
• Primary Examiner(s): Chang, Kent W
• Assistant Examiner(s): Brittingham, Nathaniel P
• Application Number:
  US15/859,457
• Publication Number:
  US 20180188894A1
• Time in Patent Office: 654 Days
• CPC Class Codes: G04G 21/08 Touch switches specially adapted for time-pieces
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/043 using propagating acoustic waves
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Systems and methods for using portions of the surface around a wearable smart device as virtual touchpads for data and command entry are disclosed. For wearable smart devices worn on the wrist like smartwatches, the skin area over the back hand as well as the surface area over the arm adjacent to the device may serve as virtual touchpads. One or more side looking cameras facing each virtual touchpad may be used to detect touches and movements by one or more fingers on the virtual touchpads to simulate equivalent tactile gestures that control the device. Various surface artifact movements captured by the cameras may be used to confirm that the finger is actually touching the virtual touchpad and not just hovering above it. Sound sensors that independently detect and locate finger touches, as well as motion sensors that detect vibration corresponding to actual touch tap events, may be combined with the cameras to improve the detection accuracy of the tactile gestures. Touch position may also be detected by triangulation of encoded RF signals that propagate from one side of the device via the operator body to the back hand virtual touchpad. Two or more virtual touchpads can operate in interchangeable or complementary modes. During interchangeable operation mode, all virtual touchpads may respond to all the tactile gestures supported by the device. In complementary mode, some virtual touchpads may be restricted to recognize only a specific subset of the supported gestures.

Claims

1. An apparatus optically interfacing with an operator responsive to a predefined set of tactile gestures performed by one or more fingers on one or more virtual touchpads assigned on portions of the surface adjacent to said apparatus, comprising:

one or more cameras placed along each side of the apparatus facing each one of said virtual touchpads, said cameras configured to obtain sequences of captured images of said virtual touchpads and their surroundings;

a display; and

a computer coupled to said cameras and said display, and configured to capture the positions of one or more finger tips and one or more visible surface artifacts that include blemishes, blood vessels, wrinkles, hair, stains, or textures on said virtual touchpads, correlate movements of said surface artifacts with said finger tip positions to distinguish between hovering and touching activities, analyze said hovering and touching activities to detect an operator tactile gesture, and to simulate, based on the detected tactile gesture, a control input to modify said display responsive to said gesture.

2. The apparatus of claim 1, further comprising one or more sound sensors configured to detect and locate tactile gestures on said virtual touchpads, and wherein said detected tactile gestures by said sound sensors are correlated and combined with said tactile gestures detected by said cameras.

3. The apparatus of claim 1, further comprising a motion sensor configured to detect operator tap events on said virtual touchpads adjacent to said apparatus, and wherein said detected tap events by said motion sensor are correlated and combined with said tactile gestures detected by said cameras.

4. The apparatus of claim 1, wherein two or more virtual touchpads are assigned on the surface adjacent to said apparatus, and wherein said virtual touchpads operate in an interchangeable operation mode wherein each virtual touchpad recognizes the entire set of predefined tactile gestures.

5. The apparatus of claim 1, wherein two or more virtual touchpads are assigned on the surface adjacent to said apparatus, and wherein said virtual touchpads operate in a complementary operation mode wherein each virtual pad is responsive only to a pre-assigned subset of said set of predefined tactile gestures.

6. The apparatus of claim 5, wherein one or more complex predefined tactile gestures comprise a combination of sub-gestures, wherein each sub-gesture is assigned to a unique virtual touchpad, and wherein said sub-gestures are performed at a predefined order.

7. The apparatus of claim 1, wherein said apparatus is a smartwatch.

8. The apparatus of claim 7, wherein a first virtual touchpad is assigned on the back hand near the wrist on which said smartwatch is worn and a second virtual touchpad is assigned on the arm adjacent to said smartwatch.

24. The apparatus of claim 1, wherein said computer is further configured to calibrate each assigned virtual touchpad by prompting the operator to tap four corners of the virtual touchpad, wherein a relative transformation is created to map touch positions to screen positions.

9. A computer-implemented method comprising:

assigning one or more adjacent portions of the body surface area of a human operator wearing a wearable device as virtual touchpads, said virtual touchpads are placed along one or more sides of said wearable device;

capturing a sequence of images of said one or more virtual touchpads;

extracting from said captured images a sequence of positions of one or more finger tips and the adjacent surface artifacts on said virtual touchpads, said surface artifacts include blood vessels, blemishes, wrinkles, hair, stains, or textures;

correlating movements of said surface artifacts with said finger tip positions to distinguish between hovering and touching activities;

analyzing said hovering and touching activities to detect a predefined gesture from a set of predefined gestures;

simulating a generic tactile touchpad input command that corresponds to said detected gesture; and

controlling said wearable device in response to said simulated tactile input command.

10. The computer-implemented method of claim 9, further comprising:

detecting finger tap positions using one or more sound sensors placed within said wearable device; and

correlating and combining said detected finger tap positions with said sequence of finger tip positions detected by said cameras.

11. The computer-implemented method of claim 9, further comprising:

detecting finger tap events from vibration data obtain by a motion sensor placed within said wearable device; and

correlating and combining said detected finger tap events with said sequence of finger tip positions detected by said cameras.

20. The computer-implemented method of claim 9, wherein assigning two or more virtual touchpads on said body surface area, further comprising operating said virtual touchpads in a complementary operation mode wherein each virtual touchpad is responsive only to a pre-assigned subset of said set of predefined gestures.

21. The computer-implemented method of claim 9, further comprising recognizing and simulating text entry inputs from actual text writing gestures made by a finger of an operator on one of said virtual touchpads.

22. The computer-implemented method of claim 9, wherein assigning two or more virtual touchpads on said body surface area, further comprising operating said virtual touchpads in an interchangeable operation mode wherein each virtual touchpad recognizes the entire set of predefined tactile gestures.

23. The computer-implemented method of claim 9, further comprising calibrating each assigned virtual touchpad by prompting the operator to tap four corners of the virtual touchpad, wherein a relative transformation is created to map touch positions to screen positions.

12. A system comprising a portable device placed on a surface, said portable device further comprising:

one or more processors;

a display;

one or more cameras coupled to said one or more processors and configured to detect one or more finger touches on one or more virtual touchpads assigned on portions of said surface, said virtual touchpads are adjacent to said portable device;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for;

capturing from said cameras a sequence of operator touch positions on said virtual touchpads;

extracting the positions of surface artifacts from images of said virtual touchpads when fingers are detected, said surface artifacts include blood vessels, stains wrinkles, hair, blemishes or textures;

correlating movements of said surface artifacts with said sequence of operator touch positions to distinguish between hovering and touching activities;

analyzing said sequence of operator touch positions to detect a predefined tactile gesture from a set of predefined tactile gestures; and

generating a tactile input command corresponding to said detected gesture to modify said display in response to the detected tactile gesture.

13. The system of claim 12, wherein said one or more cameras are placed on the side of said portable device facing each said virtual touchpad, and wherein each camera further comprising an optical system enhancing side viewing capabilities to enable image capturing of said virtual touchpad and its surrounding.

14. The system of claim 12, wherein said portable device further comprises one or more sound sensors, and wherein said one or more programs further comprising instructions for:

detecting finger tap positions using said one or more sound sensors; and

correlating and combining said detected finger tap positions with said sequence of operator touch positions detected by said cameras.

15. The system of claim 12, wherein said portable device further comprises a motion sensor, and wherein said one or more programs further comprising instructions for:

detecting finger tap events from vibration data obtain by a motion sensor placed within said portable device; and

correlating and combining said detected finger tap events with said sequence of operator touch positions detected by said cameras.

16. The system of claim 12, wherein two or more virtual touchpads are assigned on a surface adjacent to said portable device, and wherein said virtual touchpads operate in an interchangeable operation mode wherein each virtual touchpad recognizes the entire said set of predefined tactile gestures.

17. The system of claim 12, wherein two or more virtual touchpads are assigned on a surface adjacent to said portable device, and wherein said virtual touchpads operate in a complementary operation mode wherein each virtual touchpad is responsive only to a pre-assigned subset of said set of predefined tactile gestures.

18. The system of claim 12, wherein said one or more programs further comprising instructions for calibrating each assigned virtual touchpad by prompting the operator to tap four corners of the virtual touchpad, wherein a relative transformation is created to map touch positions to screen positions.

19. The system of claim 12, wherein said one or more programs further comprising instructions to recognize and simulate text entry inputs from actual text writing gestures made by a finger of an operator on one of said virtual touchpads.