Method and system enabling natural user interface gestures with an electronic system

• US 8,854,433 B1
• Filed: 02/01/2013
• Issued: 10/07/2014
• Est. Priority Date: 02/03/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Rafii, Abbas
• Primary Examiner(s): Rao, Andy
• Application Number:
  US13/757,705
• Time in Patent Office: 613 Days
• Field of Search: 348 36- 39, 348 42- 60, 348156-184
• US Class Current: 348/47
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

An electronic device coupleable to a display screen includes a camera system that acquires optical data of a user comfortably gesturing in a user-customizable interaction zone having a z₀ plane, while controlling operation of the device. Subtle gestures include hand movements commenced in a dynamically resizable and relocatable interaction zone. Preferably (x,y,z) locations in the interaction zone are mapped to two-dimensional display screen locations. Detected user hand movements can signal the device that an interaction is occurring in gesture mode. Device response includes presenting GUI on the display screen, creating user feedback including haptic feedback. User three-dimensional interaction can manipulate displayed virtual objects, including releasing such objects. User hand gesture trajectory clues enable the device to anticipate probable user intent and to appropriately update display screen renderings.

Claims

1. A method to enable a user-object to communicate using gestures made in (x,y,z) space with an electronic device coupleable to a display screen whereon imagery is displayable, the method including the following steps:

(a) providing said electronic device with an optical acquisition system operable to capture image data of said user-object within a three-dimensional hover zone;

(b) defining within said three-dimensional hover zone an interaction subzone including at least one z₀ plane disposed intermediate a plane of said display screen and a plane at a maximum z-distance beyond which user-object gestures need not be recognized by said electronic device;

(c) processing image data captured at step (a) representing an interaction of said user-object with at least a portion of said z₀ plane, defined in step (b), to detect and to produce three-dimensional positional information of said interaction;

(d) using said three-dimensional positional information produced at step (c) to determine at least one of (i) when in time, and (ii) where in (x,y,z) space said user-object interacted with said z₀ plane;

(e) following determination at step (d), identifying a gesture being made by said user-object; and

(f) in response to identification of a gesture at step (e), altering at least one operating characteristic of said electronic device.

2. The method of claim 1, wherein said user-object includes at least one of (i) at least a portion of a user'"'"'s arm, (ii) at least a portion of a user'"'"'s hand, (iii) at least a portion of a user finger, (iv) at least a portion of a user'"'"'s thumb, and (v) at least a portions of a user'"'"'s body.

3. The method of claim 1, wherein step (a) includes providing at least one optical acquisition system selected from a group consisting of:

(i) a time-of-flight (TOF) system;

(ii) a structured light system;

(iii) a single two-dimensional camera system; and

(iv) an at least two two-dimensional camera system.

4. The method of claim 1, wherein said device includes a device selected from a group consisting of (i) a smart TV, (ii) a tablet computer, (iii) a smart phone, (iv) a laptop computer, (iv) a netbook computer, (v) a device to which said display screen is not integrally attached, and (vi) an appliance.

5. The method of claim 1, wherein in response to detection of user-object interaction, step (b) includes carrying out at least one step selected from a group consisting of:

(i) customizing location of said interaction subzone and said z₀ plane within;

(ii) accommodating comfort of said user-object during interaction by dynamically altering location of said interaction subzone and said z₀ plane within;

(iii) altering size of said interaction subzone and said z₀ plane within; and

(iv) dynamically altering appearance of at least a region of imagery rendered on said display screen.

6. The method of claim 1, wherein at step (f) said electronic device creates at least one type of user feedback selected from a group consisting of (i) visual feedback presented on said display screen, (ii) acoustic feedback, and (iii) haptic feedback activating a user-object wearable haptic feedback receiving device.

7. The method of claim 1, wherein step (f) further includes at least one of (i) rendering on said display screen a GUI commensurate with said gesture identified at step (e), (ii) anticipatorily distinguishing a portion of said display screen believed to be affected by an immediately following gesture, (iii) dynamically altering size of an object rendered on said display screen, (iv) dynamically altering a viewable characteristic of an object rendered on said display screen, and (v) altering an operating mode of said device responsive to a location of said user-object detected at step (c).

8. The method of claim 1, wherein step (f) causes said electronic device to create at least one type of visual feedback selected from a group consisting of (i) displaying a ribbon menu responsive to an identified gesture that is directed at a specific region of said display screen, (ii) displaying a virtual cursor keypad responsive to an identified gesture directed at a specific region of said display screen, and (iii) displaying a dynamically size changing icon responsive to an identified gesture directed at a specific region of said display screen.

9. The method of claim 1, wherein a recognized gesture includes at least one of (i) virtually touching an object rendered on said display screen with said user-object, (ii) virtually touching an object rendered on said display screen with said user-object and altering at least one of object size and object orientation, (iii) a gesture interacting with a three-dimensional object, having at least three virtual points thereon, rendered on said display screen, each of said three virtual points mappable to one of three points on said user-object such that said user-object can virtually grasp and manipulate said object, and can then virtually decouple from and release said object.

10. The method of claim 1, wherein:

step (a) provides said optical acquisition system with at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said user-object without said user-object having to touch said display screen to interact with said z₀ plane;

step (c) processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras to carry out at least one processing step selected from a group consisting of;

(I) representing an imaged portion of said user-object user using a set of landmark points sufficient to recognize a gesture made by said user-object, and discarding at least about 90% of said number N two-dimensional data points;

(II) in addition to (I) above, further extracting from said number N of two-dimensional data points a lesser number of three-dimensional landmarks, wherein magnitude N of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude N≦

100, (ii) said magnitude N≦

50, and (iii) said magnitude N≦

12;

(III) in addition to (II) above, further processing said two-dimensional image data so as to extract from said number N of two-dimensional data points a lesser number of three-dimensional landmarks such that processing reduces acquired image information to at least one magnitude selected from a group consisting of (i-1) ≦

1% N, (ii-2) processing reduces acquired information to ≦

0.1% N, and (iii-3) processing reduces acquired information to ≦

0.01% N; and

(IV) (i-a) said processing includes edge detection, (ii-b) said processing includes object modeling, (iii-c) said processing includes image rectification, (iv-d) said processing includes epipolar geometric analysis, and (v-e) said processing includes machine learning;

wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, and (iv) said landmark is approximate location of said user'"'"'s fingertip.

21. The method of claim 1, wherein:

step (a) provides said optical acquisition system with at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said user-object without said user-object having to touch said display screen to interact with said z₀ plane;

step (c) processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras to identify potential two-dimensional locations of landmark data points on said user-object, using a relatively small number of data points typically less than 10% N; and

step (d) includes determining for at least some two-dimensional said locations identified in step (c) three-dimensional locations of potential landmarks on said user-object, wherein said determining reduces remaining image data to a sparse set of three-dimensional data typically less than 1% N.

22. The method of claim 21, wherein one of said landmarks has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, and (iv) said landmark is approximate location of said user'"'"'s fingertip.

11. A system enabling a user-object to communicate using gestures made in (x,y,z) space with an electronic device coupleable to a display screen whereon imagery is displayable, the system including:

an electronic device with an optical acquisition system operable to capture image data of said user-object within a three-dimensional hover zone;

means for defining within said three-dimensional hover zone an interaction subzone including at least one z₀ plane disposed intermediate a plane of said display screen and a plane at a maximum z-distance beyond which user-object gestures need not be recognized by said electronic device;

means for processing captured image data captured representing an interaction of said user-object with at least a portion of said z₀ plane to detect and to produce three-dimensional positional information of said interaction;

means for using processed captured said three-dimensional positional data and for determining at least one of (i) when in time, and (ii) where in (x,y,z) space said user-object first interacted with said z₀ plane;

means for identifying a gesture being made by said user-object; and

means for altering at least one operating characteristic of said electronic device responsive to gesture identification.

12. The system of claim 11, wherein said user-object includes at least one of (i) at least a portion of a user'"'"'s arm, (ii) at least a portion of a user'"'"'s hand, (iii) at least a portion of a user finger, (iv) at least a portion of a user'"'"'s thumb, and (v) at least a portions of a user'"'"'s body.

13. The system of claim 11, wherein step (a) includes providing at least one optical acquisition system selected from a group consisting of:

(i) a time-of-flight (TOF) system;

(ii) a structured light system;

(iii) a single two-dimensional camera system; and

(iv) an at least two two-dimensional camera system.

14. The system of claim 11, wherein said device includes a device selected from a group consisting of (i) a smart TV, (ii) a tablet computer, (iii) a smart phone, (iv) a laptop computer, (iv) a netbook computer, and a device to which said display screen is not integrally attached.

15. The system of claim 11, wherein in response to detection of user-object interaction, said means for defining carries out at least one step selected from a group consisting of:

(i) customizing location of said interaction subzone and said Z₀ plane within;

(ii) accommodating comfort of said user-object during interaction by dynamically altering location of said interaction subzone and said z₀ plane within;

(iii) altering size of said interaction subzone and said z₀ plane within; and

(iv) dynamically altering appearance of at least a region of imagery rendered on said display screen.

16. The system of claim 11, wherein said means for altering creates at least one type of user feedback selected from a group consisting of (i) visual feedback presented on said display screen, (ii) acoustic feedback, and (iii) haptic feedback activating a user-object wearable haptic feedback receiving device.

17. The system of claim 11, wherein said means for altering carries out at least one of (i) rendering on said display screen a GUI commensurate with said, gesture identified at step (e), (ii) anticipatorily distinguishing a portion of said display screen believed to be affected by an immediately following gesture, (iii) dynamically altering size of an object rendered on said display screen, (iv) dynamically altering a viewable characteristic of an object rendered on said display screen, and (v) altering an operating mode of said device responsive to a detected location of said user-object.

18. The system of claim 11, wherein said means for altering causes said electronic device to create at least one type of visual feedback selected from a group consisting of (i) displaying a ribbon menu responsive to an identified gesture that is directed at a specific region of said display screen, (ii) displaying a virtual cursor keypad responsive to an identified gesture directed at a specific region of said display screen, and (iii) displaying a dynamically size changing icon responsive to an identified gesture directed at a specific region of said display screen.

19. The system of claim 11, wherein said system recognizes at least one gesture selected from a group consisting of (i) virtually touching an object rendered on said display screen with said user-object, (ii) virtually touching an object rendered on said display screen with said user-object and altering at least one of object size and object orientation, (iii) a gesture interacting with a three-dimensional object, having at least three virtual points thereon, rendered on said display screen, each of said three virtual points mappable to one of three points on said user-object such that said user-object can virtually grasp and manipulate said object, and can then virtually decouple from and release said object.

20. The system of claim 11, wherein:

said optical acquisition system includes at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said user-object without said user-object having to touch said display screen to interact with said z₀ plane;

said means for processing image data processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras so as to carry out at least one processing step selected from a group consisting of;

(I) representing an imaged portion of said user-object user using a set of landmark points sufficient to recognize a gesture made by said user-object, and discarding at least about 90% of said number N data points, and representing an imaged portion of said, user-object using a set of landmark points sufficient to recognize a gesture made by said user-object;

(II) in addition to (I) above, further extracting from said number N of data points a lesser number of three-dimensional landmarks, wherein magnitude N of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude N≦

100, (ii) said magnitude N≦

50, and (iii) said magnitude N≦

12;

(III) in addition to (II) above, further processing said two-dimensional image data so as to extract from said number N of data points a lesser number of three-dimensional landmarks such that processing reduces acquired image information to at least one magnitude selected from a group consisting of (i-1) ≦

1% N, (ii-2) processing reduces acquired information to ≦

0.1% N, and (iii-3) processing reduces acquired information to ≦

0.01% N; and

(IV) (i-a) said processing includes edge detection, (ii-b) said processing includes object modeling, (iii-c) said processing includes image rectification, (iv-d) said processing includes epipolar geometric analysis, and (v-e) said processing includes machine learning;

wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is, approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, and (iv) said landmark is approximate location of said user'"'"'s fingertip.

23. The system of claim 11, wherein:

said optical acquisition system includes at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said user-object without said user-object having to touch said display screen to interact with said z₀ plane;

said means for processing image data processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras to identify potential two-dimensional locations of landmark data points on said user-object, using a relatively small number of data points typically less than 10% N;

said means for processing image data further determining for at least some two-dimensional said locations three-dimensional locations of potential landmarks on said user-object, wherein further determining reduces remaining image data to a sparse set of three-dimensional data typically less than 1% N.

24. The system of claim 23, wherein one of said landmarks has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, and (iv) said landmark is approximate location of said user'"'"'s fingertip.