Gesture recognition method and interactive input system employing same

• US 9,262,016 B2
• Filed: 01/05/2010
• Issued: 02/16/2016
• Est. Priority Date: 01/05/2009
• Status: Active Grant

• Current Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Tse, Edward, McGibney, Grant, Xu, Qizhi Joanna, Benner, Erik
• Primary Examiner(s): Moon, Seokyun
• Application Number:
  US13/143,266
• Publication Number:
  US 20120013529A1
• Time in Patent Office: 2,233 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A gesture recognition method comprises capturing images, processing the images to identify at least two clusters of touch points associated with at least two pointers, recognizing a gesture based on motion of the clusters, and updating a display in accordance with the recognized gesture.

Claims

1. A gesture recognition method comprising:

capturing images using imaging sensors having fields of view aimed generally across or at an input surface from different vantages;

processing the captured images to detect a pair of hands brought into contact with said input surface and for each detected hand calculating a bounding box, the calculated bounding box surrounding either a cluster of proximate touch points resulting from multiple fingers of the hand being in contact with said input surface or a single large touch region exceeding a threshold size resulting from a palm region of the hand being in contact with said input surface;

creating an observation for each bounding box in each captured image, each observation in each captured image defined by the area formed between two straight lines, one line of which extends from the focal point of the imaging sensor that captured the image and crosses the right edge of the bounding box and the other line of which extends from the focal point of the imaging sensor that captured the image and crosses the left edge of the bounding box;

in response to relative movement of the hands over the input surface, recognizing a gesture based on corresponding relative movement of the created observations;

executing a command associated with the recognized gesture; and

updating an image displayed on said input surface in accordance with the executed command.

3. The method of claim 1, wherein said gesture is one of a zoom gesture and a pan gesture.

4. The method of claim 3, wherein said zoom gesture is recognized by detecting an increase or a decrease in the distance between extreme edges of the created observations.

5. The method of claim 4, wherein said command is an enlarge command if the distance increases and a shrink command if the distance decreases.

6. The method of claim 3, wherein said pan gesture is recognized by detecting movement of said created observations in the same direction with the spacing between said created observations being maintained.

7. A non-transitory computer readable medium having computer program code stored thereon, the computer program code, when executed by a processor, causing an apparatus at least to perform the method of claim 1.

2. An interactive input system comprising:

an input surface;

at least two imaging sensors having fields of view aimed generally across or at said input surface from different vantages; and

processing structure communicating with said at least one imaging sensor, said processing structure being configured to;

analyze images captured by said at least one imaging sensor to detect multiple hands brought into contact with said input surface,

for each detected hand, calculate a bounding box, the bounding box surrounding either a cluster of proximate touch points resulting from multiple fingers of the hand being in contact with said input surface or a single large touch region exceeding a threshold size resulting from a palm region of the hand being in contact with said input surface,

create an observation for each bounding box in each captured image, wherein each observation in each captured image defined by the area formed between two straight lines, one line of which extends from the focal point of the imaging sensor that captured the image and crosses the right edge of the bounding box and the other line of which extends from the focal point of the imaging sensor that captured the image and crosses the left edge of the bounding box;

in response to relative movement of the hands over the input surface, recognize a gesture based on corresponding relative movement of the created bounding boxes,

execute a command associated with said recognized gesture, and

update an image displayed on said input surface in accordance with the executed command.

8. The interactive input system of claim 2, wherein said processing structure is configured to recognize one of a zoom gesture and a pan gesture.

9. The interactive input system of claim 8, wherein said processing structure is configured to recognize said zoom gesture by detecting an increase or a decrease in the distance between extreme edges of the created observations.

10. The interactive input system of claim 9, wherein said command is an enlarge command if the distance increases and a shrink command if the distance decreases.

11. The interactive input system of claim 8, wherein said processing structure is configured to recognize said pan gesture by detecting movement of said created observations in the same direction with the spacing between said created observations being maintained.

12. The interactive input system of claim 2, wherein said input surface is a display surface of a display unit.

13. The interactive input system of claim 2, wherein said input surface is a generally horizontal table surface.

14. The interactive input system of claim 13 further comprising a projector for projecting said image on said input surface.