Two-dimensional method and system enabling three-dimensional user interaction with a device

• US 9,207,773 B1
• Filed: 03/21/2014
• Issued: 12/08/2015
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Rafii, Abbas
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): Lam, Vinh
• Application Number:
  US14/222,341
• Time in Patent Office: 627 Days
• Field of Search: 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

User interaction with a display is detected substantially simultaneously using at least two cameras whose intersecting FOVs define a three-dimensional hover zone within which user interactions can be imaged. Separately and collectively image data is analyzed to identify a relatively few user landmarks. A substantially unambiguous correspondence is established between the same landmark on each acquired image, and a three-dimensional reconstruction is made in a common coordinate system. Preferably cameras are modeled to have characteristics of pinhole cameras, enabling rectified epipolar geometric analysis to facilitate more rapid disambiguation among potential landmark points. Consequently processing overhead is substantially reduced, as are latency times. Landmark identification and position information is convertible into a command causing the display to respond appropriately to a user gesture. Advantageously size of the hover zone can far exceed size of the display, making the invention usable with smart phones as well as large size entertainment TVs.

Claims

1. A method to enable a device to detect and use at least one user-made user-object interaction that represents a device-detectable event, wherein at least a portion of said user-object is representable by at least one landmark, the method including the following steps:

(a) disposing a first camera, having a first FOV (field of view), and disposing a second camera, having a second FOV, such that intersecting said first FOV and second FOV define a three-dimensional hover zone within which said user-made user-object interaction is detectable by said device;

(b) obtaining from said first camera a first two-dimensional image, comprising a first set of pixel data, of at least a portion of said user object, and obtaining from said second camera a second two-dimensional image, comprising a second set of pixel data, where a number N represents a maximum total data points acquirable by said first camera and by said second camera, wherein each said two-dimensional image is acquired within said three-dimensional hover zone;

(c) analyzing said first set and said second set of pixel data obtained at step (b) to identify therein potential two-dimensional locations of landmark data points on said user object, said analyzing reducing data to a relatively small number of data points that are typically less than 10% N;

(d) determining for at least some said two-dimensional locations identified in step (c) three-dimensional locations of potential landmarks on said user object, wherein said determining reduces remaining data to a sparse set of three-dimensional data typically less than 1% N;

(e) using three-dimensional locations determined at step (d) and using dynamic information for data remaining after step (d) to further reduce a number of three-dimensional locations of potential landmarks on said user-object by at least a factor of 10 to typically less than 0.1% N;

wherein reduced three-dimensional data following step (e) is outputtable to said device to affect at least one device parameter responsive to detected user-object interaction.

2. The method of claim 1, further including at least one limitation selected from a group consisting of:

(i) step (c) includes reducing said relatively small number of data points to less than typically about 1% N, (ii) step (d) reduces said spare set of data to less than typically about 0.1% N, and (iii) step (e) further reduces a number of three-dimensional locations of potential landmarks to less than typically about 0.1% N.

3. The method of claim 1, wherein step (c) has at least one characteristic selected from a group consisting of edge detection, and object modeling.

4. The method of claim 1, wherein step (d) includes at least one method selected from a group consisting of image rectification, and epipolar geometric analysis.

5. The method of claim 1, said user-object is a user, and said landmark includes at least one landmark selected from a group consisting of (i) approximate centroid of said user'"'"'s body, (ii) approximate centroid of said user'"'"'s head, (iii) approximate centroid of said user'"'"'s hand, (iv) approximate centroid of said user'"'"'s finger, (v) approximate location of said user'"'"'s fingertip, (vi) approximate location of said user'"'"'s shoulder joint, (vii) approximate location of said user'"'"'s knee joint, and (viii) approximate location of said user'"'"'s foot.

6. The method of claim 1, wherein:

said device is functionally coupleable to a display having a display surface; and

said user-object interaction includes at least one interaction selected from a group consisting of said user-object physically touches said display surface, and said user-object makes a gesture in a region of said three-dimensional hover zone without physically touching said display surface; and

wherein said device in response to output from step (e) creates at least one instruction usable by said device, in response to a detected said user-object interaction, said at least one instruction causing at least one action selected from a group consisting of (i) said instruction causes said device to alter at least a portion of an image presented on said display, (ii) said instruction causes said device to issue an audible sound, (iii) said instruction causes said device to alter a characteristic of said device, and (iv) said instruction causes a change in orientation in said first camera relative to said second camera.

7. The method of claim 6, wherein said device includes at least one of a gyroscope and an accelerometer, and wherein data outputtable from step (e) is modifiable by data from at least one of said gyroscope and accelerometer to alter an image on said display.

8. The method of claim 6, wherein:

said display has a characteristic selected from a group consisting of said display and said device share a common housing, and said display is separate from said device;

said first camera has at least one characteristic selected from a group consisting of (i) same pixel resolution to pixel resolution of said second camera, (ii) greater pixel resolution than pixel resolution of said second camera, (iii) same spectral sensitivity as spectral sensitivity of said second camera, and (iv) greater spectral sensitivity than spectral sensitivity of said second camera; and

said device includes at least one device selected from a group consisting of (i) a smart phone, (ii) a tablet, (iii) a netbook, (iv) a laptop, (v) an e-book reader, (vi) a PC, (vii) a TV, and (viii) a set top box.

9. The method of claim 6, wherein step (a) includes disposing said first camera and said second camera such that said three-dimensional hover zone has at least one characteristic selected from a group consisting of said three-dimensional hover zone is adjacent said surface of said display, and said three-dimensional hover zone is adjacent said surface of said display and includes at least a region of said surface of said display.

10. The method of claim 6, wherein a cross-section of said three-dimensional hover zone taken parallel to said display surface has a size that is at least as large as a diagonal dimension of said display.

11. A system to detect and use at least one user-made user-object interaction that represents a device-detectable event, wherein at least a portion of said user-object is representable by at least one landmark, the system including:

a first camera, having a first FOV (field of view), and a second camera, having a second FOV, disposed such that intersecting said first FOV and second FOV define a three-dimensional hover zone within which said user-made user-object interaction is detectable by a device;

means for obtaining a first two-dimensional image from said first camera of at least a portion of said user object within said three-dimensional hover zone, and obtaining a second two-dimensional image from said second camera having of at least a portion of said user object in said three-dimensional hover zone, wherein said first image comprises a first set of pixel data, and said second image comprises a second set of pixel data, where a number N represents a maximum total data points acquired by said first and by said second set of pixel data;

means for analyzing said first and second set of pixel data, obtained by said means for obtaining, to identify in said pixel data potential two-dimensional locations of landmark data points on said user object, such that data reduction to less than typically about 10% N occurs;

means for determining for at least some said two-dimensional locations, identified by said means for analyzing, a sparse set of three-dimensional data locations of potential landmarks on said user object, said sparse set wherein remaining data is less than typically about 1% N;

means for further reducing, coupled to use three-dimensional locations determined by said means for determining and coupled to use dynamic information for data remaining of potential landmarks to less than typically about 0.1% N;

wherein three-dimensional data obtained from said means for using is outputtable to said device to affect at least one device parameter responsive to detected user-object interaction.

12. The system of claim 11, further including at least one limitation selected from a group consisting of:

(i) said means for analyzing reduces said relatively small number of data points to less than typically about 1% N, (ii) said means for determining reduces said sparse set of data to less than typically about 0.1% N, and (iii) said means for further reducing further reduces a number of three-dimensional locations of potential landmarks to less than typically about 0.01% N.

13. The system of claim 11, wherein said means for analyzing has at least one characteristic selected from a group consisting of (i) analysis includes edge detection, and (ii) analysis includes object modeling.

14. The system of claim 11, wherein said means for determining uses at least one analytical technique selected from a group consisting of (i) image rectification, and (ii) epipolar geometric analysis.

15. The system of claim 11, wherein said-user object is a user, and said landmark includes at least one landmark selected from a group consisting of (i) approximate centroid of said user'"'"'s body, (ii) approximate centroid of said user'"'"'s head, (iii) approximate centroid of said user'"'"'s hand, (iv) approximate centroid of said user'"'"'s finger, (v) approximate location of said user'"'"'s fingertip, (vi) approximate location of said user'"'"'s shoulder joint, (vii) approximate location of said user'"'"'s knee joint, and (viii) approximate location of said user'"'"'s foot.

16. The system of claim 11, further including a display having a display surface, wherein said device is functionally coupleable to said display;

and

said user-object interaction includes at least one interaction selected from a group consisting of (i) said user-object physically touches said display surface, and (ii) a gesture made by said user-object in a region of said three-dimensional hover zone without physically touching said display surface; and

wherein said device in response to output from means for using creates at least one instruction usable by said device, in response to a detected said user-object interaction, said at least one instruction causing at least one action selected from a group consisting of (i) said instruction causes said device to alter at least a portion of an image presented on said display, (ii) said instruction causes said device to issue an audible sound, (iii) said instruction causes said device to alter a characteristic of said device, and (iv) said instruction causes a change in orientation in said first camera relative to said second camera.

17. The system of claim 16, wherein said device includes at least one of a gyroscope and an accelerometer, and wherein data outputtable from said means for further reducing is modifiable by data from at least one of said gyroscope and accelerometer to alter an image on said display;

wherein said display has a characteristic selected from a group consisting of said display and said device share a common housing, and said display is separate from said device; and

said device includes at least one device selected from a group consisting of (i) a smart phone, (ii) a tablet, (iii) a netbook, (iv) a laptop, (v) an e-book reader, (vi) a PC, (vii) a TV, and (viii) a set top box.

18. The system of claim 16, wherein said first camera and said second camera are disposed such that said three-dimensional hover zone has at least one characteristic selected from a group consisting of said three-dimensional hover zone is adjacent said surface of said display, and said three-dimensional hover zone is adjacent said surface of said display and includes at least a region of said surface of said display.

19. The system of claim 16, wherein a cross-section of said three-dimensional hover zone taken parallel to said display surface has a size that is at least as large as a diagonal dimension of said display.

20. A hand-holdable electronic device enabling at least one user-object interaction in a three-dimensional hover zone with an image presented on a display, said interaction creating a detectable event useable by said electronic device, where at least a portion of said user-object is representable by at least one landmark, the electronic device including:

a housing;

a processor-controller unit including a processor coupled to memory storing at least one routine executable by said processor, said processor-controller unit disposed in said housing;

a display having a display surface, coupled to said processor-controller unit, able to present user viewable images responsive to commands from said processor-controller unit, said display integrally joined to said housing;

at least a first camera having a first FOV and a second camera having a second FOV, said first and second camera disposed such an intersection of said first FOV and second FOV defines said three-dimensional hover zone;

said processor controller unit further including;

means for obtaining a first two-dimensional image from said first camera of at least a portion of said user object within said three-dimensional hover zone, and obtaining a second two-dimensional image from said second camera having of at least a portion of said user object in said three-dimensional hover zone, wherein said first image comprises a first set of pixel data, and said second image comprises a second set of pixel data, where a number N represents a maximum total data points acquired by said first and by said second set of pixel data;

means for analyzing said first and second set of pixel data, obtained by said means for obtaining, to identify in said pixel data potential two-dimensional locations of landmark data points on said user object, such that data reduction to substantially less than typically about 10% N occurs;

means for determining for at least some said two-dimensional locations, identified by said means for analyzing, three-dimensional locations of potential landmarks on said user object, wherein remaining data is substantially less than typically about 1% N;

means for further reducing, coupled to use three-dimensional locations determined by said means for determining and coupled to use dynamic information for data remaining of potential landmarks to less than typically about 0.1% N;

wherein three-dimensional data obtained from said means for further reducing is coupled to said electronic device to affect at least one device parameter selected from a group consisting of (i) causing said electronic device to alter at least a portion of an image presented on said display, (ii) causing said electronic device to issue an audible sound, (iii) causing said electronic device to alter a characteristic of said electronic device, and (iv) causing a change in orientation in said first camera relative to said second camera; and

wherein said electronic device includes at least one device selected from a group consisting of (i) a smart phone, (ii) a tablet, (iii) a netbook, (iv) a laptop, (v) an e-book reader, (vi) a PC, (vii) a TV, and (viii) a set top box.

21. The hand-holdable electronic device of claim 20, wherein:

said means for analyzing includes at least one analytical technique selected from a group consisting of (i) edge detection, and (ii) object modeling; and

said means for determining includes at least one analytical technique selected from a group consisting of (i) image rectification, and (ii) epipolar geometric analysis.

22. The hand-holdable electronic device of claim 20, further including at least one of a gyroscope and an accelerometer, wherein data from said means for further reducing is modifiable by data from at least one of said gyroscope and accelerometer and outputtable to said device so as to alter an image on said display.

23. The hand-holdable electronic device of claim 20, wherein said three-dimensional hover zone has at least one characteristic selected from a group consisting (i) said three-dimensional hover zone is adjacent said surface of said display, (ii) said three-dimensional hover zone is adjacent said surface of said display and includes at least a region of said surface of said display, and said three-dimensional hover zone has a cross-section taken parallel to said display surface has a size that is at least as large as a diagonal dimension of said display.