Method and system for recognition of user gesture interaction with passive surface video displays

• US 8,681,124 B2
• Filed: 09/22/2009
• Issued: 03/25/2014
• Est. Priority Date: 09/22/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bamji, Cyrus, Acharya, Sunil
• Primary Examiner(s): HALEY, JOSEPH R
• Application Number:
  US12/586,432
• Publication Number:
  US 20110291988A1
• Time in Patent Office: 1,645 Days
• Field of Search: None
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A system recognizes user-object gesture interactions with the surface of a monitor display, with hover space defined spaced-apart from the display surface, or in virtual scroll regions defined on the periphery of the monitor display. The system recognizes user-object interactions, e.g., gestures, and can affect what is displayed commensurately. The system includes at least a first time-of-flight (TOF) system and at least one of a second TOF, a two-dimensional camera, and a mirror, each TOF system processing at least one of z-depth data and A-brightness data. User-object interactions, e.g., touching(s) of the display surface, location(s) in a hover region, or location(s) in a virtual scroll region, are recognized passively in that the user-object need not have capacitance, resistance, exert force, or deform during gesture interaction. The system may be attached retroactively to the monitor, which may be a large (>22 cm) monitor, or a small cell phone sized monitor.

Claims

1. A system for use with a processor-generated display presented on a display device in a display area, the system recognizing at least one user-object interaction with a region on a surface of said display device, a hover region defined in three-dimensional space spaced-apart from said surface of said display device, and a virtual scroll region defined in three-dimensional space adjacent a peripheral region of said display area, the system including:

a first time-of-flight (TOF) system disposed to emit optical energy in a first field of view (FOV) defined generally toward said surface of said display device and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a passive user-object within said first FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one passive user-object interaction;

a second system associated with a second field of view (FOV) defined generally toward said surface of said display device, said second system including at least one of a second TOF system, at least one two-dimensional camera, and at least one mirror, wherein said first TOF system and said second system detect at least a passive user-object interaction in a location;

a system processor coupled to said first TOF system and to said second system to discern from at least one of processed z-depth data and processed A-brightness data acquired by the first TOF system and the second system the location in three-dimensional space of said at least one user-object, said location being any of (1) a portion of said surface of said display device touched by said user-object, (2) a location in said hover region, and (3) a location in a virtual scroll region, an output of said system processor coupleable to said processor-generated display to affect operation of said display on said display device.

2. The system of claim 1, wherein said second system includes second time-of-flight (TOF) system disposed to emit optical energy in a second field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said second FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction.

3. The system of claim 1, wherein said display device has a transverse dimension exceeding 22 cm.

4. The system of claim 1, wherein said system is retrofitably coupleable to said display device.

5. The system of claim 1, further including:

a library of stored gestures that a user-object may define in three-dimensional space within a FOV of at least one of said first TOF system and said second system, said library coupled to said system processor;

wherein system processor recognition of a user-object gesture detected by said system affects operation of said display on said monitor commensurate with said gesture.

6. The system of claim 1, wherein said user-object includes at least one of a user finger, a user hand, a user-manipulated stylus, a user-manipulated wand, a non-capacitive object, and a non-deformable object.

7. The system of claim 2, wherein one of said first TOF system and said second TOF system is a phase-based TOF system.

8. The system of claim 2, wherein each of said first TOF system and said second TOF system processes one type of TOF data to discern said probable location, said one type of TOF data selected from a group consisting of z-depth data, and A-brightness data.

9. The system of claim 2, wherein at least one of said first TOF system and said second TOF system emits a higher density of optical energy nearer said surface of said display than is emitted in a region spaced-apart from said surface of said display.

10. The system of claim 1, wherein user-object interaction affects operation of said display in a manner selected from a group consisting of (a) user-object manipulation of an object presented on said display, (b) user-object manipulation of operation of said display, (c) user-object manipulation with a said virtual scroll region to manipulate operation of said display, and (d) user-object manipulation in a said three-dimensional hover region to manipulate operation of said display.

11. The system of claim 1, including at least one fiducial marker disposed so as to enable self-calibration of at least said first TOF system so as to improve accuracy of data acquired therefrom.

12. The system of claim 1, further including at least one of (a) means for favoring data acquired from one of said first TOF system and said second system when it is determined that one said system is providing more reliable data, (b) means for improving data acquired by at least one of said first TOF system and said second system, (c) means for combining data from at least said first TOF system and said second system so as to improve accuracy of said system, (d) means for employing active brightness data acquired from at least said first TOF system to improve accuracy of said system, and (e) means for employing at least one of stereographic imaging, triangulation, and structured light to improve accuracy of data acquired by said system.

13. The system of claim 1, wherein said peripheral region is located adjacent to and not overlapping said display area.

14. A method, for use with a processor-generated display presented in a display area of a display device, to recognize at least one user-object interaction with at least one of region on a surface of said display device, a hover region defined in three-dimensional space spaced-apart from said surface of said display device, and a virtual scroll region defined in three-dimensional space adjacent a peripheral region of said display area, the method including the following steps:

(a) disposing a first time-of-flight (TOF) system so as to emit optical energy in a first field of view (FOV) defined generally toward said surface of said display device and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said first FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction;

(b) disposing a second system associated with a second field of view (FOV) defined generally toward said surface of said display device, said second system including at least one of a second TOF system, at least one two-dimensional camera, and at least one mirror; and

(c) discerning from at least one of processed z-depth data and processed A-brightness data from said first TOF system an actual location in three-dimensional space of said at least one user-object, said location being any of (1) a portion of said surface of said display touched by said user-object, (2) a location in said hover region, and (3) a location in said virtual scroll region, an output of said system processor coupleable to said processor-generated display to affect operation of said display on said display device.

15. The method of claim 14, wherein step (b) includes disposing a second time-of-flight (TOF) system disposed to emit optical energy in a second field of view (FOV) defined generally toward said surface of said display and to acquire at least a fraction of the emitted said optical energy that is reflected-back by a user-object within said second FOV, and to process at least one of z-depth data and A-brightness data within reflected-back said optical energy to discern a probable location in three-dimensional space of said at least one user-object interaction.

16. The method of claim 15, further including:

storing a library of stored gestures that a user-object may define in three-dimensional space within a FOV of at least one of said first TOF system and said second TOF system; and

determining from recognition of a user-object gesture detected by said system affects operation of said display on said monitor commensurate with said gesture.

17. The method of claim 14, wherein said user-object includes at least one of a user finger, a user hand, a user-manipulated stylus, a user-manipulated wand, a non-capacitive object, and a non-deformable object.

18. The method of claim 14, wherein user-object interaction affects operation of said display in a manner selected from a group consisting of (i) user-object manipulation of an object presented on said display, (ii) user-object manipulation of operation of said display, (iii) user-object manipulation with a said virtual scroll region to manipulate operation of said display, and (iv) user-object manipulation in a said three-dimensional hover region to manipulate operation of said display.

19. The method of claim 14, wherein said first TOF system emits a higher density of optical energy nearer said surface of said display than is emitted in a region spaced-apart from said surface of said display.

20. The method of claim 14, further including at least one additional step selected from a group of steps consisting of (i) providing at least one fiducial marker so as to enable self-calibration of at least said first TOF system, (ii) providing means for favoring data acquired from one of said first TOF system and said second system when it is determined that one said system is providing more reliable data, (iii) providing means for improving data acquired by at least one of said first TOF system and said second system, (iv) providing means for combining data from at least said first TOF system and said second system so as to improve accuracy of said system, (v) providing means for employing active brightness data acquired from at least said first TOF system to improve accuracy of said system, and (vi) providing means for employing at least one of stereographic imaging, triangulation, and structured light to improve accuracy of data acquired by said system.