GESTURE-BASED USER INTERFACE

• US 20150261299A1
• Filed: 07/12/2012
• Published: 09/17/2015
• Est. Priority Date: 07/12/2012
• Status: Active Grant

• Current Assignee: Dual Aperture International Co. Ltd.
• Original Assignee: Dual Aperture, Inc.
• Inventors: Wajs, Andrew Augustine
• Granted Patent: US 9,733,717 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/1353 Extracting features related to minutiae or pores
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A computer-implemented method for a gesture-based user interface and a gesture-based user interface system are described. The method comprises receiving image data from a multi-aperture image sensor in said electronic device, said image sensor being configured to simultaneously expose an image sensor to at least a first part of the electromagnetic (EM) spectrum using a first aperture and at least a second part of the EM spectrum using one or more second apertures; determining sharpness information in at least one area of said image data associated with at least part of an object imaged by said first aperture and said one or more second apertures onto the image plane of said image sensor; generating depth information on the basis of at least part of said sharpness information; and, recognizing on the basis of said depth information, at least part of a gesture associated with a movement of said object.

Claims

1. A computer-implemented method for a gesture-based user interface for controlling a software program on an electronic device, said method comprising:

receiving image data from a multi-aperture image sensor in said electronic device, said image sensor being configured to simultaneously expose an image sensor to at least a first part of the electromagnetic (EM) spectrum using a first aperture and at least a second part of the EM spectrum using one or more second apertures;

determining sharpness information in at least one area of said image data associated with at least part of an object imaged by said first aperture and said one or more second apertures onto the image plane of said image sensor;

generating depth information on the basis of at least part of said sharpness information; and

,

recognizing on the basis of said depth information, at least part of a gesture associated with a movement of said object.

2. The method according to claim 1, further comprising:

determining first image data associated with said first part of the electromagnetic spectrum;

determining second image data associated with said second part of the electromagnetic spectrum;

generating depth information by relating first sharpness information associated with at least part of an image of said object in said first image data and second sharpness information associated with at least part of said image of said object in said second image data to a distance between said image sensor and said object.

3. The method according to claim 1, wherein generating depth information comprises generating said depth information on the basis a predetermined depth function stored in a memory of said electronic device, said depth function being configured for associating sharpness information in said image data to a distance between the image sensor and the object.

4. The method according to any of claim 1 wherein generating said depth information comprises:

selecting a window in said image data;

associating sharpness information in said window to a blur kernel in a set of blur kernels on the basis of a minimization algorithm;

assigning a depth value associated with said blur kernel to one or more pixels in said window

5. The method according to claim 4 wherein said minimization algorithm comprises:

convoluting at least one of said blur kernels with said sharpness information in said window.

6. The method according to claim 1, further comprising:

determining second image data associated with said second part of the electromagnetic spectrum;

determining in said second image data at least two laterally displaced images, wherein each of said images is associated with a second aperture and wherein said images are associated with an object positioned at a distance before the image sensor;

and wherein generating depth information comprises generating depth information on the basis of a predetermined depth function stored in a memory of said electronic device, said depth function being configured for associating the distance between said laterally displaced images to a distance between said image sensor and said object.

7. The method according to claim 6, wherein said multi-aperture image sensor comprises two or more second apertures configured with respect to each other such that each of said second apertures images an object onto the image plane of said image sensor, wherein the distance between said laterally displaced images is a function of the distance between said object and said image sensor.

8. The method according to claim 6, further comprising:

determining second high-frequency image data;

determining said distance by running said second high-frequency image data through an auto-correlation function.

9. The method according to claim 1 wherein generating depth information further comprises:

forming one or more depth maps by assigning depth values to at least part of said image data generated by said multi-aperture image sensor.

10. The method according to claim 1 wherein said first part of the EM spectrum is associated with at least part of the visible part of the EM spectrum;

and/or, wherein said second part of the EM spectrum is associated with at least part of the invisible part of the EM spectrum.

11. The method according to claim 1 comprising:

determining at least part of a region of interest (ROI) in at least part of an image frame associated with said image data on the basis of said depth information.

12. The method according to claim 1 comprising:

determining at least part of a region of interest in at least part of an image frame associated with said image data on the basis of color information.

13. The method according to claim 1 wherein recognizing said gesture comprises:

extracting one or more depth features from said depth information.

14. The method according to claim 1 wherein recognizing said gesture comprises:

determining one or more trajectories;

correlating said determined trajectory with a reference trajectory associated with a predetermined gesture;

establishing the detection of a predetermined gesture if said determined one or more trajectories correlate with a one or more reference trajectories;

controlling said software program on the basis of said predetermined gesture.

15. The method according to claim 14 wherein said one or more trajectories are of one or more fingertips, said method further comprising at least one of the following:

controlling said software program on the basis of moving one or more finger tips over one or more predetermined trajectories in the field of view of said multi-aperture imaging system;

controlling said software program by moving a fingertip in a plane at a predetermined distance from said electronic device;

controlling an activation or a deactivation function in said software program on the basis of moving a fingertip from a first distance to a second distance in the field of view of said image sensor;

controlling said software program by moving said first detected finger tip relative to said second detected fingertip.

16. The method according to said method further comprising:

on the basis of said depth map determining in said image data a region of interest associated with a fingertip;

extracting one or more directional features from an enhanced image formed by blending first low-frequency image data associated with said first part of the EM spectrum with said second high-frequency image data;

authenticating a user by matching said extracted directional features with directional reference features associated with a fingerprint of said user.

17. A gesture-based user interface system for use in an electronic device, said user interface system being configured to control a software program on said electronic device, said system comprising:

a multi-aperture image sensor configured to generate image data, said multi-aperture image sensor being configured to simultaneously expose an image sensor to at least a first part of the electromagnetic (EM) spectrum using a first aperture and at least a second part of the EM spectrum using one or more second apertures;

one or more filters configured to generate sharpness information in at least one area of said image data associated with at least part of an object, preferably at least part a human body part, imaged by said first aperture and said one or more second apertures onto the image plane of said image sensor;

a generator configured to generate depth information on the basis of at least part of said sharpness information; and

,

a gesture recognition module configured to recognize on the basis of said depth information a gesture associated with a movement of said object.

18. The user interface system according to claim 17, wherein said depth function is configured to associate first sharpness information associated with said first aperture and second sharpness information associated with said one or more second apertures to a distance between the image sensor and the object;

or, wherein said depth function is configured to associate a distance between at least two laterally displaced images associated with at least two of said second apertures to a distance between said object and said image sensor.

19. The gesture-based user interface system according to claim 17, wherein said depth function is configured to associate first sharpness information associated with said first aperture and second sharpness information associated with said one or more second apertures to at least one blur kernel from a set of blur kernels stored in a memory of said electronic device, wherein each of said blur kernels is associated with a predetermined distance a distance between the image sensor and an object.

20. The user interface system according to claim 17 wherein said gesture recognition module is configured to:

determine one or more trajectories, preferably one or more spatial trajectories, of at least part of said body part on the basis said depth maps;

correlate said determined trajectory with a reference trajectory associated with a predetermined gesture;

establish the detection of a predetermined gesture if said determined one or more trajectories correlate with a one or more reference trajectories;

control said software program on the basis of said predetermined gesture.

21. The user interface system according to claim 17, wherein said gesture recognition module is configured to:

control said software program on the basis of moving one or more finger tips over one or more predetermined trajectories in the field of view of said multi-aperture imaging system;

control said software program by moving a fingertip in a plane at a predetermined distance from said electronic device;

control an activation or a deactivation function in said software program one the basis of moving a fingertip from a first distance to a second distance in the field of view of said image sensor; and

/or, control said software program by moving said first detected fingertip.

22. (canceled)