Three-dimensional touch device and method of providing the same

• US 10,452,205 B2
• Filed: 09/22/2016
• Issued: 10/22/2019
• Est. Priority Date: 04/12/2013
• Status: Active Grant

• Current Assignee: Iconics, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Iconics, Inc. (Mitsubishi Electric Corporation)
• Inventors: Kresl, Vojta, Elsbree, Christopher N., Agrusa, Russell L., De Barros, Paulo G., Altman, Petr
• Primary Examiner(s): Azari, Sepehr
• Application Number:
  US15/273,107
• Publication Number:
  US 20170010754A1
• Time in Patent Office: 1,125 Days
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

Apparatus and methods are provided for converting a display into a touch display. One or more optical sensors are arranged proximate the display. A processor receives a signal from the optical sensor(s) and, based on the signal, executes instructions for: (i) determining a distance between a pointer device and a surface of the display signal; (ii) determining a position of the pointer device on the surface of the display; (iii) when the distance is less than a threshold detection distance and greater than a threshold touch distance, providing an indication on the display that the position of the pointer device has been determined; and (iv) when the distance is less than the threshold touch distance, taking an action indicative of contact between the pointer device and the surface of the display.

Claims

1. A system comprising:

a sensor aimed in a direction substantially parallel to a surface of a display,

wherein the sensor comprises a camera and a depth sensor and is positioned outside a perimeter of the display; and

a processor configured to receive signals from the sensor and, based thereon, perform operations comprising;

determining, based on a signal from the depth sensor, a radial distance between the sensor and a pointer device disposed in front of the surface of the display;

measuring, based on a signal from the camera, a separation distance between the pointer device and the surface of the display;

adjusting the measured separation distance using a scale factor to account for the determined radial distance between the pointer device and the sensor;

determining, based on (i) the determined radial distance and (ii) the adjusted separation distance, a position of the pointer device in front of the display;

and

adjusting an image presented on the display based on the determined position.

2. The system of claim 1, wherein the depth sensor comprises a laser projector aimed in the direction substantially parallel to the surface of the display.

3. The system of claim 1, wherein determining the position comprises determining coordinates for the pointer device in three-dimensional space.

4. The system of claim 3, wherein the coordinates define a closest point to the pointer device on the surface of the display.

5. The system of claim 4, wherein the coordinates define an angular position of the pointer device, the angular position comprising an angle between (i) a centerline of the sensor along the display and (ii) a line between the sensor and the closest point.

6. The system of claim 1, wherein adjusting the image comprises determining that the separation distance is less than a threshold touch distance and, in response, providing an indication on the display that the pointer device is in contact with the surface of the display.

7. The system of claim 1, wherein adjusting the image comprises determining that the separation distance is less than a threshold detection distance and greater than a threshold touch distance and, in response thereto, providing an indication on the display that the position of the pointer device has been determined.

8. The system of claim 7, wherein providing the indication comprises modifying the image in a vicinity of the pointer device on the surface of the display.

9. The system of claim 1, wherein the pointer device comprises a finger of a user.

10. The system of claim 1, wherein determining the position of the pointer device does not require the sensor to detect a reflection of the pointer device off the surface of the display.

11. The system of claim 1, wherein determining the position of the pointer device does not require the sensor to detect a shadow of the pointer device on the surface of the display.

12. A method comprising:

receiving signals from a sensor aimed in a direction substantially parallel to a surface of a display,

wherein the sensor comprises a camera and a depth sensor and is positioned outside a perimeter of the display;

determining, based on a signal from the depth sensor, a radial distance between the sensor and a pointer device disposed in front of the surface of the display;

measuring, based on a signal from the camera, a separation distance between the pointer device and the surface of the display;

adjusting the measured separation distance using a scale factor to account for the determined radial distance between the pointer device and the sensor;

determining, based on (i) the determined radial distance and (ii) the adjusted separation distance, a position of the pointer device in front of the display;

and

adjusting an image presented on the display based on the determined position.

13. The method of claim 12, wherein the depth sensor comprises a laser projector aimed in the direction substantially parallel to the surface of the display.

14. The method of claim 12, wherein determining the position comprises determining coordinates for the pointer device in three-dimensional space.

15. The method of claim 14, wherein the coordinates define a closest point to the pointer device on the surface of the display.

16. The method of claim 15, wherein the coordinates define an angular position of the pointer device, the angular position comprising an angle between (i) a centerline of the sensor along the display and (ii) a line between the sensor and the closest point.

17. The method of claim 12, wherein adjusting the image comprises determining that the separation distance is less than a threshold touch distance and, in response, providing an indication on the display that the pointer device is in contact with the surface of the display.

18. The method of claim 12, wherein adjusting the image comprises determining that the separation distance is less than a threshold detection distance and greater than a threshold touch distance and, in response thereto, providing an indication on the display that the position of the pointer device has been determined.

19. The method of claim 18, wherein providing the indication comprises modifying the image in a vicinity of the pointer device on the surface of the display.

20. The method of claim 12, wherein the pointer device comprises a finger of a user.

21. The method of claim 12, wherein determining the position of the pointer device does not require the sensor to detect a reflection of the pointer device off the surface of the display.

22. The method of claim 12, wherein determining the position of the pointer device does not require the sensor to detect a shadow of the pointer device on the surface of the display.

23. A computer program product stored in one or more non-transitory storage media for controlling a processing mode of a computer, the computer program product being executable by the computer to cause the computer to perform operations comprising:

receiving signals from a sensor aimed in a direction substantially parallel to a surface of a display,

wherein the sensor comprises a camera and a depth sensor and is positioned outside a perimeter of the display;

determining, based on a signal from the depth sensor, a radial distance between the sensor and a pointer device disposed in front of the surface of the display;

measuring, based on a signal from the camera, a separation distance between the pointer device and the surface of the display;

adjusting the measured separation distance using a scale factor to account for the determined radial distance between the pointer device and the sensor;

determining, based on (i) the determined radial distance and (ii) the adjusted separation distance, a position of the pointer device in front of the display;

and

adjusting an image presented on the display based on the determined position.