Near touch interaction

• US 10,599,269 B2
• Filed: 08/27/2018
• Issued: 03/24/2020
• Est. Priority Date: 02/10/2011
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Imperial, Jed-Justin
• Application Number:
  US16/114,133
• Publication Number:
  US 20190012041A1
• Time in Patent Office: 575 Days
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

A near-touch interface is provided that utilizes stereo cameras and a series of targeted structured light tessellations, emanating from the screen as a light source and incident on objects in the field-of-view. After radial distortion from a series of wide-angle lenses is mitigated, a surface-based spatio-temporal stereo algorithm is utilized to estimate initial depth values. Once these values are calculated, a subsequent refinement step may be applied in which light source tessellations are used to flash a structure onto targeted components of the scene, where initial near-interaction disparity values have been calculated. The combination of a spherical stereo algorithm, and smoothing with structured light source tessellations, provides for a very reliable and fast near-field depth engine, and resolves issues that are associated with depth estimates for embedded solutions of this approach.

Claims

1. A system for determining near touch interaction, comprising:

a display for displaying one or more elements thereon;

a plurality of stereo cameras positioned adjacent to the display, at least one of the plurality of cameras being active in a visible light portion of the spectrum, and at least one of the plurality of cameras being active in a portion of the spectrum other than the visible light spectrum; and

a processor for causing the display to flash one or more visible structured light sequences on a user pointer placed adjacent to the display when ambient light is determined to be sufficient for visible spectrum viewing, and causing the display to flash one or more structured light sequences in other than the visible light portion of the spectrum when ambient light is determined to be insufficient for visibly;

spectrum viewing, determining a depth map in accordance with information acquired by the stereo camera pair, and if desired, employing the one or more structured light sequences to possibly refine the depth map, and determining a location corresponding to one or more of the one or more elements displayed on the display to be selected in accordance with the determined depth map;

wherein the at least one of the plurality of cameras that is active in the visible light portion of the spectrum operating when ambient light is determined to be sufficient for visible spectrum viewing, and wherein the at least one of the plurality of cameras that is active in other than the visible light portion of the spectrum operating when ambient light is determined to be insufficient for visible spectrum viewing.

2. The system of claim 1, further comprising one or more appropriate lenses to produce a shallow depth of field highlighting the user pointer closer to the field of view.

3. The system of claim 1, wherein the use of the shallow depth of field results in a blurred background.

4. The system of claim 1, wherein the structured light is a targeted structured light aimed at only predetermined regions of the image.

5. The system of claim 4, where in the targeted structured light is displayed on only one or more predetermined portions of the display based on the depth data, and in one or more frames.

6. The system of claim 5, wherein the one or more frames are not consecutive.

7. The system of claim 1, wherein the portion of the spectrum other than the visible light portion of the spectrum is the infrared portion of the spectrum.

8. The system of claim 1, wherein information acquired by one or more of the plurality of the cameras is dewarped.

9. The system of claim 1, wherein the processor is located remotely.

10. The system of claim 9, wherein the processor is located in a cloud computing environment.

11. A method for determining near touch interaction by a user object, comprising the steps of:

displaying on a display one or more objects;

acquiring one or more images of the user object by a plurality of cameras located adjacent to the display, at least one of the plurality of cameras being active in a visible light portion of the spectrum, and at least one of the plurality of cameras being active in a portion of the spectrum other than the visible light spectrum, wherein the one or more images is Obtained from the at least one of the plurality of cameras that is active in the visible light portion of the spectrum operating when ambient light is determined to be sufficient for visible spectrum viewing, and wherein the one or more images is obtained from the at least one of the plurality of cameras that is active in other than the visible light portion of the spectrum operating when ambient light is determined to be insufficient for visible spectrum viewing;

dewarping the one or more acquired images;

computing a disparity based upon the dewarped images;

identifying one or more target regions of the display adjacent to one or more corresponding portions of the disparity that may benefit from refinement, and displaying one or more screen tessellations therein in the visible spectrum when ambient light is determined to be sufficient for visible spectrum viewing, and displaying one or more screen tessellations therein in other than the visible light portion of the spectrum when ambient light is determined to be insufficient for visible spectrum viewing;

refining the computed disparity in accordance with additional data received in accordance with the displayed screen tessellations by at least one of the plurality of cameras that is active in the visible light portion of the spectrum operating when ambient light is determined to be sufficient for visible spectrum viewing, and by at least one of the plurality of cameras that is active in other than the visible light portion of the spectrum operating when ambient light is determined to be insufficient for visible spectrum viewing; and

determining one or more of the displayed objects to be selected by the user object in accordance with the redefined computer disparity.

12. The method of claim 11, wherein the one or more acquired images comprise a left and right image acquired at a substantially simultaneous time.

13. The method of claim 12, wherein the displaying of screen tessellations further comprises the step of flashing one or more structured light sequences on a targeted portion of the user object positioned adjacent to the display.

14. The method of claim 13, wherein the structured light is a targeted structured light.

15. The method of claim 14, where in the structured light is di splayed on only one or more predetermined portions of the display.

16. The method of claim 14, wherein the structured light emanates from the screen'"'"'s pixels.

17. The method of claim 16, wherein the screen produces patterns to refine ambiguous data in the depth map.

18. The method of claim 17, wherein the light is adaptive, such that it is brighter when ambient light is greater and darker when ambient light is less.

19. The method of claim 17, wherein the patterns that are produced vary in one or more of size, location, orientation, or frequency, depending on the size and depth of the targeted region.

20. The method of claim 11, wherein the portion of the spectrum other than the visible light portion of the spectrum is the infrared portion of the spectrum.

21. The method of claim 11, w-herein the tessellations are interleaved with the actual data, allowing the display of the structured light without the light being perceived by the user.

22. The method of claim 11, wherein the one or more cameras comprise lenses that are able to properly display objects that are closer to the cameras in the field of view.

23. The method of claim 11, wherein the one or more cameras comprise lenses that play the role of multiple bandpass filters to allow and block different frequencies of the lighting spectrum.