FLEXIBLE AIR AND SURFACE MULTI-TOUCH DETECTION IN MOBILE PLATFORM

• US 20150309663A1
• Filed: 11/18/2014
• Published: 10/29/2015
• Est. Priority Date: 04/28/2014
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Seo, Hae-Jong, Wyrwas, John Michael, Maitan, Jacek, Gousev, Evgeni Petrovich, Aryan, Babak, Cui, Xiquan
• Application Number:
  US14/546,303
• Publication Number:
  US 20150309663A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

Systems, methods, and apparatus for recognizing user interactions with an electronic device are provided. Implementations of the systems, methods, and apparatus include surface and air gesture recognition and identification of fingertips or other objects. In some implementations, a device including a plurality of detectors configured to receive signals indicating interaction of an object with the device at or above a detection area, such that a low resolution image can be generated from the signals, is provided. The device is configured to obtain low resolution image data from the signals and obtain a first reconstructed depth map from the low resolution image data. The first reconstructed depth map may have a higher resolution than the low resolution image. The device is further configured to obtain a second reconstructed depth map from the first reconstructed depth map. The second reconstructed depth map may provide improved boundaries and less noise within the object.

Claims

1. An apparatus comprising:

an interface for a user of an electronic device having a front surface including a detection area;

a plurality of detectors configured to detect interaction of an object with the device at or above the detection area and output signals indicating the interaction, wherein an image can be generated from the signals; and

a processor configured to;

obtain image data from the signals;

apply a linear regression model to the image data to obtain a first reconstructed depth map, wherein the first reconstructed depth map has a higher resolution than the image; and

apply a trained non-linear regression model to the first reconstructed depth map to obtain a second reconstructed depth map.

2. The apparatus of claim 1, further comprising one or more light-emitting sources configured to emit light, wherein the plurality of detectors are light detectors and the signals indicate interaction of the object with light emitted from the one or more light-emitting sources.

3. The apparatus of claim 1, further comprising:

a planar light guide disposed substantially parallel to the front surface of the interface, the planar light guide including;

a first light-turning arrangement that is configured to output reflected light, in a direction having a substantial component orthogonal to the front surface, by reflecting emitted light received from one or more light-emitting sources; and

a second light-turning arrangement that redirects light resulting from the interaction toward the plurality of detectors.

4. The apparatus of claim 1, wherein the second reconstructed depth map has a resolution at least three times greater than the resolution of the image.

5. The apparatus of claim 1, wherein the second reconstructed depth map has the same resolution as the first reconstructed depth map.

6. The apparatus of claim 1, wherein the processor is configured to recognize, from the second reconstructed depth map, an instance of a user gesture.

7. The apparatus of claim 6, wherein the interface is an interactive display and wherein the processor is configured to control one or both of the interactive display and the electronic device, responsive to the user gesture.

8. The apparatus of claim 1, wherein the apparatus does not have a time-of-flight depth camera.

9. The apparatus of claim 1, wherein obtaining image data comprises vectorization of the image.

10. The apparatus of claim 1, wherein obtaining a first reconstructed depth map includes applying a learned weight matrix to vectorized image data to obtain a first reconstructed depth map matrix.

11. The apparatus of claim 1, wherein apply a non-linear regression model to the first reconstructed depth map includes extracting a multi-pixel patch feature for each pixel of the first reconstructed depth map to determine a depth map value for each pixel.

12. The apparatus of claim 1, wherein the object is a hand.

13. The apparatus of claim 12, wherein the processor is configured to apply a trained classification model to the second reconstructed depth map to determine locations of fingertips of the hand.

14. The apparatus of claim 13, wherein the locations include translation and depth location information.

15. The apparatus of claim 1, wherein the object is a stylus.

16. An apparatus comprising:

an interface for a user of an electronic device having a front surface including a detection area;

a plurality of detectors configured to receive signals indicating interaction of an object with the device at or above the detection area, wherein an image can be generated from the signals; and

a processor configured to;

obtain image data from the signals;

obtain a first reconstructed depth map from the image data, wherein the first reconstructed depth map has a higher resolution than the image; and

apply a trained non-linear regression model to the first reconstructed depth map to obtain a second reconstructed depth map.

17. The apparatus of claim 16, further comprising one or more light-emitting sources configured to emit light, wherein the plurality of detectors are light detectors and the signals indicate interaction of the object with light emitted from the one or more light-emitting sources.

18. The apparatus of claim 16, further comprising:

a planar light guide disposed substantially parallel to the front surface of the interface, the planar light guide including;

a first light-turning arrangement that is configured to output reflected light, in a direction having a substantial component orthogonal to the front surface, by reflecting emitted light received from one or more light-emitting sources; and

a second light-turning arrangement that redirects light resulting from the interaction toward the plurality of detectors.

19. A method comprising:

obtaining image data from a plurality of detectors arranged along a periphery of a detection area of a device, the image data indicating an interaction of an object with the device at or above the detection area;

obtaining a first reconstructed depth map from the image data, wherein the first reconstructed depth map has a higher resolution than the image; and

obtaining a second reconstructed depth map from the first reconstructed depth map.

20. The method of claim 19, wherein obtaining the first reconstructed depth map includes applying a learned weight matrix to vectorized image data.

21. The method of claim 20, further comprising learning the weight matrix.

22. The method of claim 21, wherein learning the weight matrix includes obtaining training set data of pairs of depth maps and images for multiple object gestures and positions, wherein the resolution of the depth maps is higher than the resolution of the images.

23. The method of claim 19, wherein obtaining a second reconstructed depth map includes applying a non-linear regression model to the first reconstructed depth map.

24. The method of claim 23, wherein applying a non-linear regression model to the first reconstructed depth map includes extracting a multi-pixel patch feature for each pixel of the first reconstructed depth map to determine a depth map value for each pixel.

25. The method of claim 24, further comprising learning the non-linear regression model.

26. The method of claim 19, wherein the second reconstructed depth map has a resolution at least three times greater than the resolution of the image.

27. The method of claim 19, wherein the object is a hand.

28. The method of claim 27, further comprising applying a trained classification model to the second reconstructed depth map to determine locations of fingertips of the hand.

29. The method of claim 28, wherein the locations include translation and depth location information.