Gesture recognition interface system with vertical display

• US 8,180,114 B2
• Filed: 06/05/2008
• Issued: 05/15/2012
• Est. Priority Date: 07/13/2006
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Nishihara, H. Keith, Hsu, Shi-Ping
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): Bitar, Nancy
• Application Number:
  US12/133,836
• Publication Number:
  US 20080244468A1
• Time in Patent Office: 1,440 Days
• Field of Search: 382/112, 382/113, 382/114, 382/288, 382/103, 382/106, 382/228, 345/863, 345/158, 345/156, 345/358, 715/863
• US Class Current: 382/114
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

One embodiment of the invention includes a gesture recognition interface system. The system may comprise a substantially vertical surface configured to define a gesture recognition environment based on physical space in a foreground of the substantially vertical surface. The system may also comprise at least one light source positioned to provide illumination of the gesture recognition environment. The system also comprises at least two cameras configured to generate a plurality of image sets based on the illumination being reflected from an input object in the gesture recognition environment. The system further comprises a controller configured to determine a given input gesture based on changes in relative locations of the input object in each of the plurality of image sets. The controller may further be configured to initiate a device input associated with the given input gesture.

Claims

1. A gesture recognition interface system comprising:

a substantially vertical surface in which three-dimensional physical space in a foreground of the substantially vertical surface defines a gesture recognition environment;

at least one light source positioned to provide illumination of the gesture recognition environment between the at least one light source and the substantially vertical surface;

at least two cameras configured to generate a plurality of image sets based on the illumination that is provided from the at least one light source being reflected from an input object positioned in the gesture recognition environment; and

a controller configured to implement an algorithm that detects features associated with the input object in each of the plurality of image sets based on determining a sequence of mid-points between opposing edges of elongated portions of the input object in a predetermined coordinate system in each of the plurality of image sets and based on correlating the sequence of mid-points of each of a first image with corresponding mid-points of a second image in each of the plurality of image sets to calculate endpoints, length, and pitch of the elongated portions of the input object and to determine a given input gesture performed in the gesture recognition environment based on changes in relative locations of the input object in each of the plurality of image sets, and being further configured to initiate a device input associated with the given input gesture.

2. The interface system of claim 1, wherein the at least two cameras are positioned such that a central lens axis of each of the at least two cameras is substantially parallel with respect to a plane that is defined by the substantially vertical surface to capture the substantially vertical surface and the gesture recognition environment in the plurality of image sets.

3. The interface system of claim 1, wherein the at least one light source is positioned such that a central illumination axis of the at least one light source has a non-zero angle of incidence with respect to a normal associated with a floor under the substantially vertical surface.

4. The interface system of claim 1, wherein the substantially vertical surface is configured to display visual content that is interactive with the input object.

5. The interface system of claim 4, wherein the controller is configured to provide the visual content as virtual ink on portions of the substantially vertical surface that are contacted by the input object.

6. The interface system of claim 5, wherein at least one of a color change, a pen-width change, and erasure of the virtual ink is controlled via a respective at least one additional predetermined gesture.

7. The interface system of claim 1, wherein the at least one light source comprises at least one infrared (IR) light source, and wherein the at least two cameras each comprise an IR filter.

8. The interface system of claim 7, wherein the controller comprises an automated calibration unit configured to calibrate the gesture recognition interface system based on an amount of parallax separation between matched sets of images of the input object that is placed in contact with at least one predetermined two-dimensional location on the substantially vertical surface.

9. The interface system of claim 1, wherein the substantially vertical surface is configured as a substantially vertical retroreflective surface to substantially mitigate the receipt of illumination reflected from the input object to the substantially vertical surface by at least one of the at least two cameras.

10. A method of providing device inputs, the method comprising:

displaying visual content on a substantially vertical display surface;

illuminating a gesture recognition environment that is defined by a three-dimensional physical space in a foreground of the substantially vertical display surface via at least one light source;

providing simulated inputs that are interactive with the visual content in the gesture recognition environment via gestures associated with an input object;

generating a first plurality of images and a second plurality of images associated with the input object substantially concurrently based on reflected light from the input object that is provided from the at least one light source;

detecting features associated with the input object in each of corresponding pairs of the first and second plurality of images based on determining edges of the input object in each of the first and second plurality of images based on the reflected light and based on generating a sequence of mid-points between opposing edges along the elongated portions of the input object in a predetermined coordinate system in each of the first and second plurality of images;

determining a plurality of three-dimensional physical locations of the input object based on a relative separation of the features associated with the input object in the corresponding pairs of the first and second plurality of images;

determining if changes in the plurality of three-dimensional physical locations of the input object correspond to any of a plurality of pre-defined gestures; and

providing at least one device input based on a given one of the plurality of pre-defined gestures upon determining that the changes in the plurality of three-dimensional physical locations of the input object correspond to the given one of the plurality of pre-defined gestures.

11. The method of claim 10, further comprising:

contacting at least one predetermined point in two-dimensional space on the substantially vertical display surface with an endpoint of the input object;

generating a first and second calibration image of the input object in contact with the at least one predetermined point; and

calibrating the three-dimensional physical locations of the input object based on an amount of parallax separation between the endpoint of the input object in each of the first and second calibration image.

12. A gesture recognition interface system comprising:

means for providing a brightness contrast between reflected light and non-reflected light in a gesture recognition environment that is defined by a three-dimensional physical space in front of a substantially vertical display surface, the reflected light being reflected from an input object in the gesture recognition environment;

means for generating a first plurality of images of the input object based on the brightness contrast;

means for generating a second plurality of images of the input object based on the brightness contrast, the first plurality of images and the second plurality of images forming a plurality of matched pairs of images of the input object;

means for generating a sequence of points associated with elongated portions of the input object in each of the first and second plurality of images and for correlating at least one pair of corresponding points in each of the plurality of matched pairs of images of the input object to calculate endpoints, length, and pitch of the elongated portions of the input object;

means for determining three-dimensional physical location information associated with features of the input object based on a relative separation of the correlated at least one pair of corresponding points in the plurality of matched pairs of images of the input object;

means for translating changes in the three-dimensional location information associated with the features of the input object to a given input gesture; and

means for providing device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures.

13. The interface system of claim 12, wherein the means for generating the sequence of points is configured to determine edges of the input object in each of the first and second plurality of images based on the brightness contrast and to generate a sequence of mid-points between opposing edges of elongated portions of the input object in a predetermined coordinate system in each of the first and second plurality of images.

14. The interface system of claim 12, wherein the means for generating the first plurality of images and the means for generating the second plurality of images are configured to obtain the first and second plurality of images at a glancing angle that is substantially parallel with respect to a plane that is defined by the substantially vertical display surface to capture the substantially vertical display surface and the gesture recognition environment in each of the first and second plurality of images.

15. The interface system of claim 12, further comprising means for calibrating the means for determining three-dimensional location information associated with the input object based on an amount of parallax separation of a matched pair of images of an endpoint of the input object in contact with at least one predetermined two-dimensional physical location on the substantially vertical display surface.