Method and system for recognition of user gesture interaction with passive surface video displays

• US 20110291988A1
• Filed: 09/22/2009
• Published: 12/01/2011
• Est. Priority Date: 09/22/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Bamji, Cyrus, Acharya, Sunil
• Granted Patent: US 8,681,124 B2
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A system recognizes user-object gesture interactions with the surface of a monitor display, with hover space defined spaced-apart from the display surface, or in virtual scroll regions defined on the periphery of the monitor display. The system recognizes user-object interactions, e.g., gestures, and can affect what is displayed commensurately. The system includes at least a first time-of-flight (TOF) system and at least one of a second TOF, a two-dimensional camera, and a mirror, each TOF system processing at least one of z-depth data and A-brightness data. User-object interactions, e.g., touching(s) of the display surface, location(s) in a hover region, or location(s) in a virtual scroll region, are recognized passively in that the user-object need not have capacitance, resistance, exert force, or deform during gesture interaction. The system may be attached retroactively to the monitor, which may be a large (>22 cm) monitor, or a small cell phone sized monitor.

Claims

1. A system for use with a processor-generated display presented on a monitor, the system recognizing at least one user-object interaction with at least one of a region on a surface of said display, a hover region defined in three-dimensional space spaced-apart from said surface of said display, and a virtual scroll region defined in three-dimensional space adjacent a peripheral region of said display, the system including:

a first time-of-flight (TOF) system disposed to emit optical energy in a first field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said first FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction;

a second system associated with a second field of view (FOV) defined generally toward said surface of said display, said second system including at least one of a second TOF system, at least one two-dimensional camera, and at least one mirror;

a system processor coupled to said first TOF system and to said second system to discern from at least one of processed z-depth data and processed A-brightness data an actual location in three-dimensional space of said at least one user-object, said location being at least one of a portion of said surface of said display touched by said user-object, a location in said hover region, and a location in a virtual scroll region, an output of said system processor coupleable to said processor-generated display to affect operation of said display on said monitor.

2. The system of claim 1, wherein said second system includes a second time-of-flight (TOF) system disposed to emit optical energy in a second field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said second FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction

3. The system of claim 1, wherein said display has a transverse dimension exceeding 22 cm,

4. The system of claim 1, wherein said system is retrofitably coupleable to said monitor.

5. The system of claim 1, further including:

a library of stored gestures that a user-object may define in three-dimensional space within a FOV of at least one of said first TOF system and said second system, said library coupled to said system processor;

wherein system processor recognition of a user-object gesture detected by said system affects operation of said display on said monitor commensurate with said gesture.

6. The system of claim 1, wherein said user-object includes at least one of a user finger, a user hand, a user-manipulated stylus, a user-manipulated wand, a non-capacitive object, and a non-deformable object.

7. The system of claim 2, wherein one of said first TOF system and said second TOF system is a phase-based TOF system.

8. The system of claim 2, wherein each of said first TOF system and said second TOF system processes one type of TOF data to discern said probable location, said one type of TOF data selected from a group consisting of z-depth data, and A-brightness data.

9. The system of claim 2, wherein at least one of said first TOF system and said second TOF system emits a higher density of optical energy nearer said surface of said display than is emitted in a region spaced-apart from said surface of said display.

10. The system of claim 1, wherein user-object interaction affects operation of said display in a manner selected from a group consisting of (a) user-object manipulation of an object presented on said display, (b) user-object manipulation of operation of said display, (c) user-object manipulation with a said virtual scroll region to manipulate operation of said display, and (d) user-object manipulation in a said three-dimensional hover region to manipulate operation of said display.

11. The system of claim 1, including at least one fiducial marker disposed so as to enable self-calibration of at least said first TOF system so as to improve accuracy of data acquired therefrom.

12. The system of claim 1, further including at least one of (a) means for favoring data acquired from one of said first TOF system and said second system when it is determined that one said system is providing more reliable data, (b) means for improving data acquired by at least one of said first TOF system and said second system, (c) means for combining data from at least said first TOF system and said second system so as to improve accuracy of said system, (d) means for employing active brightness data acquired from at least said first TOF system to improve accuracy of said system, and (e) means for employing at least one of stereographic imaging, triangulation, and structured light to improve accuracy of data acquired by said system.

13. The system of claim 1, wherein said display is on a monitor in a system selected from a group consisting of (a) a computer display, (b) a kiosk display, (c) a TV display, (d) a PDA display, and (e) a cell phone display.

14. A method, for use with a processor-generated display presented on a monitor, to recognize at least one user-object interaction with at least one of region on a surface of said display, a hover region defined in three-dimensional space spaced-apart from said surface of said display, and a virtual scroll region defined in three-dimensional space adjacent a peripheral region of said display, the method including the following steps:

(a) disposing a first time-of-flight (TOF) system so as to emit optical energy in a first field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said first FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction;

(b) disposing a second system associated with a second field of view (FOV) defined generally toward said surface of said display, said second system including at least one of a second TOF system, at least one two-dimensional camera, and at least one mirror; and

(c) discerning from at least one of processed z-depth data and processed A-brightness data from said first TOF system an actual location in three-dimensional space of said at least one user-object, said location being at least one of a portion of said surface of said display touched by said user-object, a location in said hover region, and a location in a virtual scroll region, an output of said system processor coupleable to said processor-generated display to affect operation of said display on said monitor.

15. The method of claim 14, wherein step (b) includes disposing a second time-of-flight (TOF) system disposed to emit optical energy in a second field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said second FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction

16. The method of claim 15, further including:

storing a library of stored gestures that a user-object may define in three-dimensional space within a FOV of at least one of said first TOF system and said second TOF system; and

determining from recognition of a user-object gesture detected by said system affects operation of said display on said monitor commensurate with said gesture.

17. The method of claim 14, wherein said user-object includes at least one of a user finger, a user hand, a user-manipulated stylus, a user-manipulated wand, a non-capacitive object, and a non-deformable object.

18. The method of claim 14, wherein user-object interaction affects operation of said display in a manner selected from a group consisting of (i) user-object manipulation of an object presented on said display, (ii) user-object manipulation of operation of said display, (iii) user-object manipulation with a said virtual scroll region to manipulate operation of said display, and (iv) user-object manipulation in a said three-dimensional hover region to manipulate operation of said display.

19. The method of claim 14, wherein said first TOF system emits a higher density of optical energy nearer said surface of said display than is emitted in a region spaced-apart from said surface of said display.

20. The method of claim 14, further including at least one additional step selected from a group of steps consisting of (i) providing at least one fiducial marker so as to enable self-calibration of at least said first TOF system, (ii) providing means for favoring data acquired from one of said first TOF system and said second system when it is determined that one said system is providing more reliable data, (iii) providing means for improving data acquired by at least one of said first TOF system and said second system, (iv) providing means for combining data from at least said first TOF system and said second system so as to improve accuracy of said system, (v) providing means for employing active brightness data acquired from at least said first TOF system to improve accuracy of said system, and (vi) providing means for employing at least one of stereographic imaging, triangulation, and structured light to improve accuracy of data acquired by said system.