Optical surface using a reflected image for determining three-dimensional position information

• US 9,195,344 B2
• Filed: 03/23/2011
• Issued: 11/24/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Neonode, Inc.
• Original Assignee: Neonode, Inc.
• Inventors: Goertz, Magnus, Eriksson, Thomas, Shain, Joseph, Jansson, Anders, Kvist, Niklas, Pettersson, Robert, Sparf, Lars, Karlsson, John
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): CHOWDHURY, AFROZA Y
• Application Number:
  US13/069,404
• Publication Number:
  US 20110169782A1
• Time in Patent Office: 1,707 Days
• Field of Search: 345/169, 345/173, 345/179
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

A touch screen system including a reflective display surface, a camera mounted so as to capture an image of (i) the reflective display surface, (ii) a pointer approaching the reflective display surface, and (iii) a reflection of the pointer on the reflective display surface, and a processor coupled with the camera that determines a three-dimensional location of the pointer relative to the reflective display surface, based on the positions of the pointer and the reflection of the pointer in the image captured by the camera.

Claims

1. A camera-based touch sensor comprising:

a glossy surface that reflects light in a mirror-like fashion and upon which a user performs input gestures to control an electronic device;

a camera aimed at least at a portion of said surface along a line of vision such that when a pointer is near said surface the camera captures a two-dimensional pixel image of the pointer and a reflection of the pointer by said surface, the pointer having a pixel location (px, py) and the reflection of the pointer having a pixel location (rx, ry) in the pixel image; and

a processor coupled with said camera configured to determine a three-dimensional location (X,Y, Z) of the pointer relative to said surface, by;

determining a two-dimensional location (X, Y) of the projection of the pointer onto said surface, by (i) projecting (px, py) in the pixel image onto a first axis that is perpendicular to said camera'"'"'s line of vision, and (ii) projecting the mid-point between (px, py) and (rx, ry) in the pixel image onto a second axis that is parallel to said camera'"'"'s line of vision, and

determining a height, Z, of the pointer above said surface based on the distance between (px, py) and (rx, ry) in the pixel image.

2. The touch sensor of claim 1, whereby said processor determines a height Z=0 based on coincidence of (px, py) and (rx, ry).

3. The touch sensor of claim 1 wherein said surface comprises a computer screen.

4. The touch sensor of claim 1 wherein said camera is rigidly mounted on a side of said surface.

5. The touch sensor of claim 1 wherein said camera is adjustably mounted on a side of said surface.

6. The touch sensor of claim 5 wherein said processor calibrates said adjustably mounted camera with said surface by correlating known locations on said surface that are being touched by the pointer, with their corresponding locations within one or more pixel images of said surface captured by said camera.

7. The touch sensor of claim 6 wherein said processor ascertains that the pointer and the reflection of the pointer are at a common position within the one or more pixel images of said surface captured by said camera, prior to the correlating.

8. A method for determining a location of a pointer by a touch sensor, comprising:

providing a glossy surface that reflects light in a mirror-like fashion and upon which a user performs input gestures to control an electronic device;

capturing, by a camera aimed at least at a portion of the surface along a line of vision, a two-dimensional pixel image including a pointer and a reflection of the pointer by the surface, the pointer being near the surface, the pointer having a pixel location (px, py) and the reflection of the pointer having a pixel location (rx, ry) in the pixel image; and

determining a three-dimensional location (X,Y, Z) of the pointer relative to the surface, comprising;

determining a two-dimensional location (X, Y) of the projection of the pointer onto the surface, comprising;

projecting (px, py) in the pixel image onto a first axis that is perpendicular to the camera'"'"'s line of vision; and

projecting the mid-point between (px, py) and (rx, ry) in the pixel image onto a second axis that is parallel to the camera'"'"'s line of vision; and

determining a height, Z, of the pointer above the surface based on the distance between (px, py) and (rx, ry) in the pixel image.

9. The method of claim 8 wherein said determining a height comprises determining a height Z=0 based on coincidence of (px, py) and (rx, ry).

10. The method of claim 8 wherein the camera is adjustably mounted on the surface, the method further comprising calibrating the camera by correlating known locations on the surface that are being touched by the pointer, with their corresponding locations within one or more pixel images of the surface captured by the camera.

11. The method of claim 10 further comprising ascertaining that the pointer and the reflection of the pointer are at a common position in the one or more pixel images of the surface captured by the single camera, prior to said correlating.