Enhanced interaction touch system

• US 10,324,566 B2
• Filed: 12/11/2017
• Issued: 06/18/2019
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: FlatFrog Laboratories AB
• Original Assignee: FlatFrog Laboratories AB
• Inventors: Christiansson, Tomas
• Primary Examiner(s): Kohlman, Christopher J
• Application Number:
  US15/837,558
• Publication Number:
  US 20180129354A1
• Time in Patent Office: 554 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A touch-sensitive apparatus includes an touch sensing part which operates a touch sensor arrangement to provide signal data representing a touch within a touch-sensing region on a front surface of a light transmissive panel; The touch-sensitive apparatus further includes a computer vision system part which operates a camera system to image a scene located externally of the touch-sensitive apparatus, and operates a computer vision controller to detect, based on image data generated by the camera system, at least one object within the scene. The touch-sensitive apparatus enables user interaction by touch control, gesture control and hover control.

Claims

1. A touch-sensitive apparatus, comprising:

a touch surface;

a touch sensor arrangement configured to be responsive to touches within a touch-sensing region on the touch surface;

a touch controller configured to determine a location of a touch within the touch-sensing region based on signal data from the touch sensor arrangement;

a camera system configured to image a scene located external to the touch-sensitive apparatus; and

a computer vision controller configured to detect at least one object within the scene based on image data generated by the camera system,

discriminate between an object spaced from the touch-sensing region and an object in contact with the touch-sensing region, and

output a position of a touch of the object on the touch surface when the object is in contact with the touch-sensing region and output a hover control where the object is spaced from the touch-sensing region,

wherein the touch-sensitive apparatus is further configured to match a position of the object in contact with the touch-sensing region, as determined by the computer vision controller, with the location of the touch in the touch-sensing region, as determined by the touch controller, and

wherein the touch-sensitive apparatus is further configured to assign a unique identifier to at least one of the at least one object or a user of the at least one object, said user being detected by the computer vision controller.

2. The touch-sensitive apparatus of claim 1, wherein the computer vision controller is further configured to determine a position of the at least one object in three dimensions external to the touch-sensitive apparatus.

3. The touch-sensitive apparatus of claim 2, wherein

the computer vision controller is further configured to match the position of the at least one object to one or more control spaces with an extent external to the touch-sensitive apparatus, and

said one or more control spaces include a proximity control space, which overlaps at least part of the touch-sensing region and extends no more than 50 mm from the touch surface.

4. The touch-sensitive apparatus of claim 3, wherein the computer vision controller is further configured to associate a movement pattern of the at least one object within the proximity control space with a hover gesture.

5. The touch-sensitive apparatus of claim 3, wherein said one or more control spaces include a gesture control space, which is spaced from the touch surface by at least 50 mm.

6. The touch-sensitive apparatus of claim 5, wherein

the computer vision controller is further configured to associate a movement pattern of the at least one object within the gesture control space with a control gesture for controlling operation of the touch-sensitive apparatus, and

the control gesture is associated with one of

controlling a power setting of the touch-sensitive apparatus, and

interacting with a user interface generated by a display device beneath the touch surface.

7. The touch-sensitive apparatus of claim 3, wherein the touch-sensitive apparatus is further configured to set a power consumption of at least one of the touch controller and the touch sensor arrangement based on the position of the at least one object in relation to the one or more control spaces.

8. The touch-sensitive apparatus of claim 3, wherein the hover control comprises at least the position of the object within the proximity control space.

9. The touch-sensitive apparatus of claim 1, wherein the computer vision controller is further configured to

determine a movement pattern of the at least one object within the scene,

match the movement pattern to a set of gestures, and

generate a gesture command associated with a gesture among the set of gestures.

10. The touch-sensitive apparatus of claim 9, wherein the computer vision controller is further configured to at least one of determine the movement pattern or match the movement pattern to the set of gestures, provided that the at least one object fulfills at least one constraint defined by at least one of a size, a position, an orientation, a movement history, an object count, or an inter-object spacing.

11. The touch-sensitive apparatus of claim 1, wherein the computer vision controller is further configured to

detect that the at least one object is moving towards the touch surface, and

disable a process of detecting gestures based on movement of the at least one object when the at least one object is moving towards the touch surface.

12. The touch-sensitive apparatus of claim 11, wherein the touch-sensitive apparatus is further configured to increase power consumption of at least one of the touch controller and the touch sensor arrangement when the computer vision controller is further configured to detect that the at least one object is moving towards the touch surface.

13. The touch-sensitive apparatus of claim 1, wherein the touch-sensitive apparatus is further configured to decrease power consumption of at least one of the camera system and the computer vision controller if the touch controller indicates presence of a touch within the touch-sensing region.

14. The touch-sensitive apparatus of claim 1, wherein the touch sensor arrangement comprises:

light emitters and light detectors along a perimeter of the touch-sensing region, the light emitters configured to generate light beams that propagate across the touch-sensing region above the touch surface on a plurality of propagation paths to the light detectors, the touch-sensitive apparatus configured such that a touch on the touch surface within the touch-sensing region causes attenuation of light received by the light detectors on one or more of the plurality of propagation paths.

15. The touch-sensitive apparatus of claim 14, wherein

the light emitters and the light detectors are arranged along a border region surrounding the touch-sensing region, as seen in a direction normal to a rear of the touch surface, and

the camera system is arranged within said border region, as seen in the direction normal to the rear of the touch surface.

16. The touch-sensitive apparatus of claim 15, wherein

the border region includes an optical filter, which is configured to transmit at least a portion of light emitted by the light emitters while blocking visible light, and the camera system is optically coupled to a rear of the touch surface via the optical filter.

17. A method of operating a touch-sensitive apparatus, said method comprising:

operating a touch sensor arrangement to provide signal data representing a touch within a touch-sensing region on a touch surface of a panel;

operating a touch controller to determine a location of the touch within the touch-sensing region based on the signal data;

operating a camera system to image a scene located external to the touch-sensitive apparatus; and

operating a computer vision controller to

detect at least one object within the scene based on image data generated by the camera system,

discriminate between an object spaced from the touch-sensing region and an object in contact with the touch-sensing region, and

output a position of a touch of the object on the touch surface when the object is in contact with the touch-sensing region and output a hover control where the object is spaced from the touch-sensing region,

matching a position of the object in contact with the touch-sensing region, as determined by the computer vision controller, with the location of the touch in the touch-sensing region, as determined by the touch controller, and

assigning a unique identifier to at least one of the at least one object or a user of the at least one object, said user being detected by the computer vision controller.

18. A non-transitory, tangible computer readable storage medium comprising program instructions which, when executed by one or more processors, cause the one or more processors to carry out the method of claim 17.