Computer vision gesture based control of a device

• US 8,666,115 B2
• Filed: 10/13/2010
• Issued: 03/04/2014
• Est. Priority Date: 10/13/2009
• Status: Expired due to Fees

• Current Assignee: PointGrab Ltd.
• Original Assignee: PointGrab Ltd.
• Inventors: Perski, Haim, Kaplan, Amir, Eilat, Eran
• Primary Examiner(s): Newman, Michael A
• Application Number:
  US13/499,929
• Publication Number:
  US 20120200494A1
• Time in Patent Office: 1,238 Days
• Field of Search: 382/103, 382/181, 382/218
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method are provided for controlling a device based on computer vision. Embodiments of the system and method of the invention are based on receiving a sequence of images of a field of view; detecting movement of at least one object in the images; applying a shape recognition algorithm on the at least one moving object; confirming that the object is a user hand by combining information from at least two images of the object; and tracking the object to control the device.

Claims

1. A method for computer vision based hand gesture device control, the method comprising

receiving a sequence of images of a field of view, said images comprising at least one object;

detecting movement of the at least one object in the images, to define at least one moving object;

applying a shape recognition algorithm on the at least one moving object in each of at least two of the sequence of images, to obtain shape information;

performing a first stage determination that the at least one moving object is a user hand by determining that the movement of the at least one object is a movement in a predefined pattern;

performing a second stage determination that the at least one moving object is a user hand by combining the shape information from each of the at least two of the sequence of images to obtain a combined shape information, and determining that the combined shape information corresponds to a hand shape; and

subject to successful first and second stage determination, tracking the user hand to control the device.

2. The method according to claim 1 comprising tracking the user hand to identify at least one control gesture.

3. The method according to claim 1, wherein the combined shape information comprises a shape affinity grade associated with said object in each image.

4. The method according to claim 3 wherein the combined shape information is a function of at least two shape affinity grades from at least two images.

5. The method according to claims 4 wherein combining information from at least two images of the object comprises calculating an average of the shape affinity grades from at least two images.

6. The method according to claim 5 comprising comparing the average to a predetermined threshold and deeming the second stage determination positive if the average is above the threshold.

7. The method according to claim 1 comprising:

determining a shape recognition threshold;

following the determination in the first stage that the movement of the at least one object is a movement in a predefined pattern, lowering the shape recognition threshold, such that the longer the movement is in a predefined pattern, the lower the shape recognition threshold is.

8. The method according to claim 2 wherein the movement in predefined pattern is an initializing gesture and wherein the initializing gesture and the at least one control gesture are not the same.

9. The method according to claim 8 wherein the initializing gesture is a wave-like movement of the hand.

10. The method according to claim 2 wherein the at least one control gesture is to switch a system to cursor control mode.

11. The method according to claim 1 wherein the combined shape information corresponds to a hand shape only if it is a hand with extended fingers.

12. The method according to claim 1 comprising:

receiving images of a field of view;

tracking the movement of more than one object in the images; and

comparing the movement of at least one of the objects to a predefined movement pattern; and

if at least one object is moving in a predefined movement pattern, applying shape recognition algorithms to confirm a shape of a hand.

13. The method according to claim 1 wherein tracking comprises selecting points of interest within one image in the sequence of images, detecting the location of the point in another image and identifying points with similar movement and location parameters.

14. The method according to claim 1 comprising applying an edge detection algorithm to confirm the object is a user hand.

15. The method according to claim 1 comprising deriving 3D information from the received sequence of images and combining the 3D information with the information combined from at least two images.

16. A system for user-device interaction, the system comprising:

an electronic device, an image sensor and a processor, the processor to;

receive a sequence of images of a field of view;

detect a movement of at least one moving object in the images, to define at least one moving object;

apply a shape recognition algorithm on the at least one moving object in each of at least two of the sequence of images, to obtain shape information;

perform a first stage determination that the at least one moving object is a user hand by determining that the movement of the at least one object is a movement in a predefined pattern;

perform a confirmatory second stage determination that the at least one moving object is a user hand by combining the shape information from each of the at least two of the sequence of images to obtain a combined shape information dataset, and comparing the shape information dataset against a reference hand shape information dataset; and

subject to successful first and second stage determination, track the object to control the device.

17. The system according to claim 16 wherein the processor is to track the object to identify at least one control gesture and to generate a command to the device based on the at least one control gesture.

18. The system according to claim 16 wherein the image sensor is a 3D image sensor.

19. The system according to claim 16 comprising one or more 2D image sensors.