Method and system enabling natural user interface gestures with user wearable glasses

• US 9,310,891 B2
• Filed: 09/03/2014
• Issued: 04/12/2016
• Est. Priority Date: 09/04/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Rafii, Abbas, Zuccarino, Tony
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US14/476,649
• Publication Number:
  US 20150062003A1
• Time in Patent Office: 587 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

User wearable eye glasses include a pair of two-dimensional cameras that optically acquire information for user gestures made with an unadorned user object in an interaction zone responsive to viewing displayed imagery, with which the user can interact. Glasses systems intelligently signal process and map acquired optical information to rapidly ascertain a sparse (x,y,z) set of locations adequate to identify user gestures. The displayed imagery can be created by glasses systems and presented with a virtual on-glasses display, or can be created and/or viewed off-glasses. In some embodiments the user can see local views directly, but augmented with imagery showing internet provided tags identifying and/or providing information as to viewed objects. On-glasses systems can communicate wirelessly with cloud servers and with off-glasses systems that the user can carry in a pocket or purse.

Claims

1. A method to enable an unadorned user-object to communicate using gestures made in (x,y,z) space with an eye glasses wearable electronic device coupleable to a display having a display screen whereon user viewable imagery is displayable, the eye glasses having an optical acquisition system operable to capture image data, the method comprising:

(a) capturing image data of said unadorned user-object within a three-dimensional hover zone;

(b) defining within said three-dimensional hover zone an interaction subzone including at least one z₀ plane disposed intermediate said eye glasses wearable electronic device and a plane at a maximum z-distance beyond which unadorned user-object gestures need not be recognized by said electronic device;

(c) processing image data captured at (a) representing an interaction of said unadorned user-object with at least a portion of said interaction subzone, defined in (b), to produce three-dimensional positional information of a detected said interaction, the processing including transforming the image data from coordinates corresponding to the optical acquisition system to world coordinates;

(d) using said three-dimensional positional information produced at (c) to determine at least one of (i) when in time, and (ii) where in (x,y,z) space said unadorned user-object interaction occurred;

(e) following determination at (d), identifying a gesture being made by said unadorned user-object; and

(f) in response to identification of a gesture at (e), generating and coupling at least one command to said display, said command having at least one characteristic selected from a group consisting of (I) said command causes altering at least one aspect of said viewable imagery, and (II) said command causes alteration of a state of said display regardless of whether an altered said state is user viewable.

2. The method of claim 1, wherein said eye glasses wearable electronic device is provided with at least one of (i) a video generation system generating said viewable imagery, and (ii) said display screen, mechanically coupled to said eye glasses wearable electronic device, whereon said viewable imagery is displayed for viewing by said user.

3. The method of claim 1, wherein said viewable imagery is displayed on a device external to said eye glasses wearable electronic device.

4. The method of claim 1, wherein (a) includes providing an at least two two-dimensional camera system.

5. The method of claim 1, wherein at (c) said unadorned user-object interaction occurs within at least a portion of said z₀ plane.

6. The method of claim 1, wherein in response to detection of unadorned user-object interaction, (b) includes carrying out at least one selected from a group consisting of:

(i) customizing location of said interaction subzone and said z₀ plane within;

(ii) accommodating comfort of said unadorned user-object during interaction by dynamically altering location of said interaction subzone and said z₀ plane within;

(iii) altering size of said interaction subzone and said z₀ plane within; and

(iv) dynamically altering appearance of at least a region of imagery rendered on said display screen.

7. The method of claim 1, wherein (f) further includes at least one of (i) rendering a cursor object targeting a GUI element on said display screen, (ii) rendering on said display screen a GUI commensurate with said gesture identified at (e), (iii) anticipatorily distinguishing a portion of said display screen believed to be affected by an immediately following gesture, (iv) dynamically altering size of an object rendered on said display screen, (v) dynamically altering a viewable characteristic of an object rendered on said display screen, and (vi) altering an operating mode of said eye glasses wearable electronic device responsive to a location of said unadorned user-object detected at (c).

8. The method of claim 1, wherein (f) causes said eye glasses wearable electronic device to create at least one type of visual feedback, presented on said display screen, selected from a group consisting of (i) displaying a ribbon menu responsive to an identified gesture that is directed at a specific region of said display screen, (ii) displaying a virtual cursor keypad responsive to an identified gesture directed at a specific region of said display screen, and (iii) displaying a dynamically size changing icon responsive to an identified gesture directed at a specific region of said display screen.

9. The method of claim 1, wherein a recognized gesture includes at least one of (i) virtually touching an object rendered on said display screen with said unadorned user-object, (ii) virtually touching an object rendered on said display screen with said unadorned user-object and altering at least one of object size and object orientation, (iii) a gesture interacting with a three-dimensional object, having at least three virtual points thereon, rendered on said display screen, each of said three virtual points mappable to one of three points on said unadorned user-object such that said unadorned user-object can virtually grasp and manipulate said object, and can then virtually decouple from and release said object, (iv) generating and coupling a command commensurate to said recognized gesture to said display.

10. The method of claim 1, wherein:

(a) provides said optical acquisition system with at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said unadorned user-object without said unadorned user-object having to touch said display screen for detectable interaction to occur;

(c) processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras to carry out at least one processing act selected from a group consisting of;

(I) representing an imaged portion of said unadorned user-object user using a set of landmark points sufficient to recognize a gesture made by said unadorned user-object, and discarding at least about 90% of said number N data points, and representing an imaged portion of said unadorned user-object using a set of landmark points sufficient to recognize a gesture made by said unadorned user-object;

(II) in addition to (I) above, further extracting from said number N of data points a lesser number of three-dimensional landmarks, wherein magnitude N of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude N≦

100, (ii) said magnitude N≦

50, and (iii) said magnitude N≦

12;

(III) in addition to (II) above, further processing said two-dimensional image data so as to extract from said number N of data points a lesser number of three-dimensional landmarks such that processing reduces acquired image information to at least one magnitude selected from a group consisting of (i-1) ≦

1% N, (ii-2) processing reduces acquired information to ≦

0.1% N, and (iii-3) processing reduces acquired information to ≦

0.01% N; and

(IV) (i-a) said processing includes edge detection, (ii-b) said processing includes object modeling, (iii-c) said processing includes image rectification, (iv-d) said processing includes epipolar geometric analysis, and (v-e) said processing includes machine learning; and

wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s hand, (ii), said landmark is approximate centroid of said user'"'"'s finger, and (iii) said landmark is approximate location of said user'"'"'s fingertip.

11. An eye glasses wearable system enabling an unadorned user-object to communicate using gestures made in (x,y,z) space with an eye glasses wearable electronic device coupleable to a display having a display screen whereon user viewable imagery is displayable, the system including:

an eye glasses system including an optical acquisition system operable to capture image data of said unadorned user-object within a three-dimensional hover zone, said optical acquisition system including at least two two-dimensional cameras;

a processor; and

memory storing instructions that, when executed by the processor, cause the processor to;

define within said three-dimensional hover zone an interaction subzone including at least one z₀ plane disposed intermediate a plane of said display screen and a plane at a maximum z-distance beyond which unadorned user-object gestures need not be recognized by said electronic device;

process captured image data representing an interaction of said unadorned user-object with at least a portion of said interaction subzone to produce three-dimensional positional information of a detected said interaction, the processor further transforming the image data from coordinates corresponding to the optical acquisition system to world coordinates;

determine at least one of (i) when in time, and (ii) where in (x,y,z) space said unadorned user-object first interaction occurred based on the three-dimensional positional information;

identify a gesture being made by said unadorned user-object; and

generate and couple at least one command to said display, said command having at least one characteristic selected from a group consisting of (I) said command causes altering at least one aspect of said viewable imagery, and (II) said command causes alteration of a state of said display regardless of whether an altered said state is user viewable.

12. The eye glasses wearable system of claim 11, further including at least one of (i) a video generation system generating said viewable imagery, (ii) said display and said display screen, mechanically coupled to said eye glasses wearable electronic device, whereon said viewable imagery is displayed for viewing by said user, and (iii) means for attaching said eye glasses wearable system retrofittably to an existing pair of user wearable eye glasses.

13. The eye glasses wearable system of claim 11, wherein said unadorned user-object includes at least one of (i) at least a portion of a user'"'"'s arm, (ii) at least a portion of a user'"'"'s hand, (iii) at least a portion of a user finger, and (iv) at least a portion of a user'"'"'s thumb.

14. The eye glasses wearable system of claim 11, wherein said unadorned user-object interaction occurs within at least a portion of said z₀ plane.

15. The eye glasses wearable system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to, in response to detection of user-object interaction, carries out at least one selected from a group consisting of:

(i) customizing location of said interaction subzone and said z₀ plane within;

(ii) accommodating comfort of said unadorned user-object during interaction by dynamically altering location of said interaction subzone and said z₀ plane within;

(iii) altering size of said interaction subzone and said z₀ plane within; and

(iv) dynamically altering appearance of at least a region of imagery rendered on said display screen.

16. The eye glasses wearable system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to, when generating and coupling, carries out at least one of (i) rendering a cursor object targeting a GUI element on said display screen, (ii) rendering on said display screen a GUI commensurate with an identified said gesture, (iii) anticipatorily distinguishing a portion of said display screen believed to be affected by an immediately following gesture, (iv) dynamically altering size of an object rendered on said display screen, (v) dynamically altering a viewable characteristic of an object rendered on said display screen, and (vi) altering an operating mode of said device responsive to a detected location of said unadorned user-object.

17. The eye glasses wearable system of claim 11, wherein said eye glasses wearable system recognizes at least one gesture selected from a group consisting of (i) virtually touching an object rendered on said display screen with said unadorned user-object, (ii) virtually touching an object rendered on said display screen with said unadorned user-object and altering at least one of object size and object orientation, (iii) a gesture interacting with a three-dimensional object, having at least three virtual points thereon, rendered on said display screen, each of said three virtual points mappable to one of three points on said unadorned user-object such that said unadorned user-object can virtually grasp and manipulate said object, and can then virtually decouple from and release said object.

18. The eye glasses wearable system of claim 11, wherein said eye glasses wearable electronic device further includes at least one of (i) a video generation system generating said viewable imagery, and (ii) said display screen, mechanically coupled to said eye glasses wearable electronic device, whereon said viewable imagery is displayed for viewing by said user.

19. The eye glasses wearable system of claim 11, wherein said viewable imagery is displayed on a device external to said eye glasses wearable electronic device.

20. The eye glasses wearable system of claim 11, wherein:

said optical acquisition system includes at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone;

said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said unadorned user-object without said unadorned user-object having to touch said display screen to interact with said z₀ plane;

the memory further stores instructions that, when executed by the processor, cause the processor to process two-dimensional image data acquired by said two spaced-apart two-dimensional cameras so as to carry out at least one processing act selected from a group consisting of;

(I) representing an imaged portion of said unadorned user-object user using a set of landmark points sufficient to recognize a gesture made by said unadorned user-object, and discarding at least about 90% of said number N data points, and representing an imaged portion of said unadorned user-object using a set of landmark points sufficient to recognize a gesture made by said unadorned user-object;

(II) in addition to (I) above, further extracting from said number N of data points a lesser number of three-dimensional landmarks, wherein magnitude N of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude N≦

100, (ii) said magnitude N≦

50, and (iii) said magnitude N≦

12;

(III) in addition to (II) above, further processing said two-dimensional image data so as to extract from said number N of data points a lesser number of three-dimensional landmarks such that processing reduces acquired image information to at least one magnitude selected from a group consisting of (i-1) ≦

1% N, (ii-2) processing reduces acquired information to ≦

0.1% N, and (iii-3) processing reduces acquired information to ≦

0.01% N; and

(IV) (i-a) said processing includes edge detection, (ii-b) said processing includes object modeling, (iii-c) said processing includes image rectification, (iv-d) said processing includes epipolar geometric analysis, and (v-e) said processing includes machine learning; and

wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, and (iv) said landmark is approximate location of said user'"'"'s fingertip.