METHODS AND SYSTEMS FOR CONTROLLING DEVICES USING GESTURES AND RELATED 3D SENSOR

• US 20130009861A1
• Filed: 05/23/2012
• Published: 01/10/2013
• Est. Priority Date: 07/04/2011
• Status: Active Grant

• Current Assignee: 3divi Company
• Original Assignee: 3DIVI
• Inventors: Valik, Andrey, Zaitsev, Pavel, Morozov, Dmitry
• Granted Patent: US 8,823,642 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Provided are computer-implemented methods and systems for controlling devices using gestures and a 3D sensor that enables implementing the above. In one embodiment, the method proposed herein may be based on defining at least one sensor area within the space surrounding a user of a controlled device; associating this sensor area with at least one user gesture; associating the combination of the user gesture and sensor area with an actionable command; identifying the direction of the line of sight of the user and a focal point of the line of sight of the user; and, if the line of sight of the user is directed a sensor area, issuing an actionable command corresponding to the combination of the sensor area and the gesture that the user makes while looking at this sensor area.

Claims

1. A method for controlling a device using gestures, the method comprising:

capturing a three-dimensional (3D) image associated with a user of the device;

identifying whether a line of site of the user is directed towards a predetermined area in a space surrounding the user, the predetermined area being a sensor area, the line of sight being an imaginary straight line between a position of eyes of the user and an object that the user views;

identifying a gesture made by the user substantially contemporaneously with the current line of sight;

comparing the gesture to one or more predetermined gestures associated with the sensor area; and

based on the comparison, selectively providing to the device a command corresponding to the combination of the sensor area and the gesture.

2. The method of claim 1, wherein the direction of the line of sight is determined based on one or more of the following:

position of the eyes of the user, an orientation of a head of the user, a position of a hand of a user, an angle of inclination of the head, and a rotation of the head.

3. The method of claim 1, wherein the line of sight is a beam crossing a point associated with a hand of the user.

4. The method of claim 1, wherein the direction of the line of sight is calculated based on a position of the pupils and/or a contour of the irises of the eyes.

5. The method of claim 1, wherein determination of the line of sight is based on a predetermined manual or automatic calibration of a 3D sensor being used to capture the 3D image.

6. The method of claim 1, wherein a center of one of the eyes of the user is taken as the position of eyes.

7. The method of claim 1, wherein a point on an interval connecting centers of eyes of the user is taken as the position of eyes.

8. The method of claim 1, wherein the sensor area is substantially flat and is disposed within a same plane as a display communicatively coupled to the controlled device.

9. The method of claim 8, wherein the sensor area is located within the display.

10. The method of claim 9, wherein a pointer is displayed on the display and moved on the display in accordance with motions of a hand or a pointing device.

11. The method of claim 9, wherein a pointer is displayed on the display and moved in accordance with motions of a focal point of the vision line.

12. The method of claim 8, wherein the sensor area is located outside the display.

13. The method of claim 8, wherein the sensor area is adjacent with a boundary of the display.

14. The method of claim 1, further comprising determining a focal point of the vision line and tracking the focal point.

15. The method of claim 14, wherein a movement of the focal point from outside to inside the display, a graphic object is displayed on the display.

16. The method of claim 15, wherein a horizontal movement of a hand directed to the display displaying a number of graphical objects is associated with a scrolling command.

17. The method of claim 1, wherein the gesture is recognized based on an analysis of fingers of the user.

18. The method of claim 17, wherein bending of fingers previously unbent and directed toward the sensor area connected to an object on a display is associated with a command to select an object.

19. The method of claim 17, wherein opening a palm, previously clenched in a fist or a pinch, towards an object on a display, is associated with a command to maximize the object.

20. The method of claim 1, wherein a position and a movement of a pointing device held by the user is analyzed with any elongated object being used as the pointing device.

21. The method of claim 20, wherein single or multiple touches by a finger of the pointing device held by the rest of the fingers of the same hand are recognized.

22. The method of claim 1, wherein a face is recognized and the user is identified based on the 3D image.

23. A system for controlling a device using gestures, the system comprising:

an infrared (IR) projector to generate modulated light;

an IR camera to capture a three-dimensional (3D) image associated with a user of the device; and

a computing unit communicatively coupled to the IR projector and the IR camera, the computing unit being configured to analyze the 3D image and, based on the analysis, to;

identify a sensor area within a direction of a current line of sight of the user, the line of sight being an imaginary straight line between a position of eyes of the user and an object that the user views,

identify a gesture made by the user substantially contemporaneously with the current line of sight,

compare the gesture to one or more predetermined gestures associated with the sensor area; and

based on the comparison, selectively provide to the device a command corresponding to the combination of the sensor area and the gesture.

24. The system of claim 23, wherein in order to analyze the 3D image, the computing unit is configured to:

filter the 3D image;

receive a map of folded phases;

transform the map of folded phases into a map unfolded phases;

calculate a map of depths; and

calculate 3D coordinates of parts of a body of the user.

25. The system of claim 23, further comprising a color video camera.

26. The system of claim 25, wherein the IR projector, color video camera, and IR camera are installed in a common housing.

27. The system of claim 25, wherein the color video camera and/or the IR camera are equipped with liquid lenses.

28. The system of claim 23, wherein the computing unit is equipped with an interface to transmit a video signal to a display.