Apparatus and system for interfacing with computers and other electronic devices through gestures by using depth sensing and methods of use

• US 9,041,775 B2
• Filed: 03/23/2012
• Issued: 05/26/2015
• Est. Priority Date: 03/23/2011
• Status: Active Grant

• Current Assignee: Mgestyk Technologies Inc.
• Original Assignee: Mgestyk Technologies Inc.
• Inventors: Ionescu, Dan, Ionescu, Bogdan, Islam, Shahidul M., Gadea, Cristian, Suse, Viorel
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US13/428,727
• Publication Number:
  US 20120242800A1
• Time in Patent Office: 1,159 Days
• Field of Search: 348/46
• US Class Current: 348/46
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

Disclosed herein are systems and methods for gesture capturing, detection, recognition, and mapping them into commands which allow one or many users to interact with electronic games or any electronic device interfaces. Gesture recognition methods, apparatus and system are disclosed from which application developers can incorporate gesture-to-character inputs into their gaming, learning or the like applications. Also herein are systems and methods for receiving 3D data reflecting hand, fingers or other body parts movements of a user, and determining from that data whether the user has performed gesture commands for controlling electronic devices, or computer applications such as games or others.

Claims

1. An apparatus for 3D mapping images of one or more objects, comprising:

a. an illumination device having a radiation source which emits radiation pulses varying in frequency, duty cycle and intensity, for irradiating one or more objects present in the field of view (“

FOV”

), wherein the frequency, duty cycle and intensity are adjusted by the illumination device and wherein the radiation pulses are incoherent and homogeneous radiation;

b. an optical device for projecting one or more object images onto an image sensing device;

c. the image sensing device;

d. a reconfigurable architecture hardware comprising an image capturing device for capturing images of one or more objects from the image sensing device, an information storage device, the reconfigurable hardware for controlling the illumination device and implementing a 3D mapping algorithm generating a 3D map of one or more objects; and

e. a processor comprising an information storage device, for processing the 3D map of one or more objects.

2. The apparatus of claim 1 wherein the illumination device spatially samples the scene in the FOV by gradually irradiating with radiation one or more objects placed in the FOV of the optical device, and is controlled by the reconfigurable architecture hardware.

3. The apparatus of claim 2 wherein the radiation pulses are unobstructed.

4. The apparatus of claim 1, wherein the reconfigurable architecture hardware further comprises a communication device, the hardware implementing the depth image calculation to calculate a depth field of one or more objects, and the hardware implementing a finite state machine which controls both the image sensing device and the illumination device, and the hardware implementing a second finite state machine for communication with the processor and the external devices.

5. The apparatus of claim 1 further comprising a housing, wherein the illumination device, the optical device, the image sensing device, the reconfigurable architecture hardware and the processor are contained within the housing.

6. The apparatus of claim 1 wherein a plurality of radiation pulses form an image sensor device frame, and the plurality of radiation pulses differ from each other in frequency from frame to frame.

7. The apparatus of claim 6 wherein the plurality of radiation pulses differ from each other in duty cycle and intensity from frame to frame.

8. The apparatus of claim 1 wherein the frequency varies in discrete values.

9. The apparatus of claim 1 wherein the radiation source is a laser and the illumination device comprises a disperser to disperse the radiation pulses uniformly.

11. The system of claim 10 further comprising a movement controller, the hardware of which produces signals which measure the movement of hands or other body parts in motion, based on acceleration and rotation of the body parts movement, and which can be read by a processor for building data to implement a “

six degrees of freedom”

control of external electronic devices, comprising;

a. an apparatus of claim 1;

b. a 3D mapping of one or more objects;

c. a controller which comprises acceleration and rotation transducers; and

d. a method for fusing the 3D map with the data produced by the controller which comprises acceleration and rotation transducers the combination of the two data sets producing a combined data set permitting thus the control of the environment in 6 degrees of freedom.

10. A system for using a 3D mapping image of one or more objects for controlling electronic devices, comprising:

a. an illumination device having a radiation source which emits radiation pulses varying in frequency, duty cycle and intensity for irradiating one or more objects in the FOV, wherein the radiation pulses are incoherent and homogeneous radiation;

b. an optical device for projecting one or more object images onto an image sensing device;

c. the image sensing device which produces images based on the reflection of the variable light radiation of one or more objects situated in the FOV of the optical device;

d. a reconfigurable hardware architecture controlling the illumination device comprising an image capturing device for capturing images of one or more objects object from the image sensing device devices, implementing a 3D mapping formula for generating a 3D map of the one or more objects present in the FOV; and

e. a processor for processing the 3D map of one or more objects for analyzing gestures in order to interpret them through special image processing methods for mapping them into desired commands to control electronic devices.

12. The system of claim 10 further comprising a plurality of systems communicating over a network allowing two or more system users to interact in controlling electronic devices such that the users can collaborate on controlling in concert or competitively the electronic device or the application which executes on that electronic device.

13. The system of claim 10 further comprising a communication device for sending the result of the processing of the 3D map to the controlled device, the controlled device being any device which can be controlled locally or remotely through a communication protocol specific to the electronic device.

14. A method of irradiating one or more objects comprising the steps of:

a. controlling an illumination device to project a controlled train of radiation pulses that are varying in frequency, duty cycle and intensity onto the one or more objects, wherein the radiation pulses are incoherent and homogeneous radiation;

b. adapting the pulses to the distance of one or more objects from the center of the optical device; and

c. controlling image sensing device parameters according to parameters of the varying in frequency and duty-cycle pulses.

15. The method of claim 14 further comprising the steps of:

d. establishing illumination device parameters by providing a 3D map to a processor and receiving from the processor the illumination device parameters needed;

e. using data obtained from the processor for calculating parameters of the depth map calculation algorithm; and

f. establishing communication with the processor and with an electronic device.

16. The method of claim 14 further comprising the steps of:

d. detecting and recognizing representative features of a face of one or more objects where the object placed in front of the imaging device is a human; and

e. identifying the face of one or more humans present in the field of view and using the identified face of the human to further process other parts of interest from the body of the human object.

17. The method of claim 16 further comprising the processing of the depth images for identifying body or component parts of one or more objects comprising the steps of:

f. obtaining depth image data from a reconfigurable architecture hardware;

g. processing the data to obtain characteristic features;

h. applying corresponding calculation methods identifying body or component parts of one or more objects for real-time matching and tracking of gestures and body parts posture or movements; and

i. mapping a meaning of the gesture or body parts posture or movements onto an action to be transmitted to an external electronic device.

18. The method of claim 16 further comprising the steps of:

f. identifying a sequence of gestures made by using hands and fingers;

g. identifying and separating right and left hand wrists;

h. identifying one or more fingers by applying adequate image processing methods, such as building convex, or concave polygons, or applying classification methods or alike;

i. labeling each finger;

j. tracking the fingers;

k. identifying and interpreting a hand and finger sequence of gesture; and

l. mapping the meaning of the hand and finger gesture onto an interaction with an electronic device.

19. The method of claim 16 further comprising the steps of:

f. Identifying a sequence of gestures made by either using hands, fingers or body or body parts posture or movements to create a meaning for any gesture made by the user;

g. using the face identification, the body or body parts identification or the hand and finger identification results to detect, identify, and interpret the sequence of gestures a gesture which is a combination of face, body or body parts, and hand and fingers movements; and

h. mapping the result of identifying the meaning of the sequence of gestures gesture or body parts movements onto an action to be transmitted to the external electronic device.

20. The method of claim 19 further comprising the steps of:

i. recognizing the sequence of gestures gesture made by one or more objects as part of a gesture language stored in an information storage device of the processor;

j. having defined a gesture language made out by a sequence of gestures, each of which represents the meaning of a natural language word or a phrase, or a sequence of gesture statements, sequence which can be either predefined or assembled by a user of the system;

k. parsing gestures stored in the information storage of the processor which interprets the gestures defined by the gesture language and assembles them into a command to be sent to electronic devices connected to the system; and

l. assembling the gesture language out of the predefined gestures representing cues or from user-defined gestures by means of a graphical user interface.

21. The method of claim 20 further comprising the step of:

m. allowing the user to define its own sequence of gestures gesture or body parts movements comprising the steps of;

i. detecting, identifying, interpreting, and extracting the cue of the new sequence of gestures gesture as defined by the user;

ii. integrating the new sequence of gestures gesture in the language parser; and

iii. storing the new gesture in information storage device.