Wearable device with intelligent user-input interface

• US 9,857,919 B2
• Filed: 07/12/2016
• Issued: 01/02/2018
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Original Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Inventors: Zhang, Wei
• Primary Examiner(s): ELAHI, SHAN E
• Application Number:
  US15/207,502
• Publication Number:
  US 20160320919A1
• Time in Patent Office: 539 Days
• Field of Search: 348 61
• CPC Class Codes: G01B 11/022 by means of tv-camera scanning
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

A wearable device, having a camera and a light source, receives user inputs by a first method of measuring a height of a finger-like object'"'"'s tip above a reference surface if this surface is present, or a second method of estimating a 3D location of the tip. In the first method, a plain sheet of light is projected to the object, casting a shadow on the surface. A camera-observed shadow length is used to compute the tip'"'"'s height above the surface. In the second method, the nearest and farthest locations of the tip are estimated according to pre-determined lower and upper bounds of the object'"'"'s physical width. The object is then illuminated with a structured-light pattern configured such that an area between the nearest and farthest locations receives a portion of the pattern where this portion does not contain a repeated sub-pattern, enabling unique determination of the tip'"'"'s 3D location.

Claims

1. A method for receiving user inputs by a wearable device, the wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, the method comprising:

when a finger-like object having a tip is detected within a field of view (FOV) of the camera, determining a three-dimensional (3D) location of the tip so that one or more of the user inputs are determinable according to the 3D location of the tip;

wherein the determining of the 3D location of the tip comprises;

capturing a first image containing at least the tip in a presence of the plain sheet of light illuminating the object;

from the first image, determining an on-sensor location of the tip, and an on-sensor length of a width of the object;

estimating a nearest physical location and a farthest physical location of the tip according to a pre-determined lower bound and a pre-determined upper bound of the object'"'"'s physical width, respectively, and further according to the on-sensor location of the tip, the on-sensor length of the object'"'"'s width, and the focal length, whereby the tip is estimated to be physically located within a region-of-presence between the nearest and farthest physical locations;

projecting the structured-light pattern to at least the region-of-presence such that a part of the object around the tip is illuminated with a first portion of the structured-light pattern while the region-of-presence receives a second portion of the structured-light pattern, wherein the light source configures the structured-light pattern to have the second portion not containing any repeated sub-pattern, enabling unique determination of the 3D location of the tip by uniquely identifying the first portion of the structured-light pattern inside the second portion of the structured-light pattern;

capturing a second image containing at least the part of the object around the tip when the structured-light pattern is projected to the region-of-presence; and

determining the 3D location of the tip by identifying, from the second image, the first portion of the structured-light pattern.

2. The method of claim 1, wherein the estimating of the nearest and farthest physical locations of the tip comprises:

computing a direction vector pointing the on-sensor location of the tip from the optical center of the camera;

estimating a lower bound and an upper bound of an object-camera distance according to the pre-determined lower bound and the pre-determined upper bound of the object'"'"'s physical width, respectively, and further according to the focal length, and the on-sensor length of the object'"'"'s physical width; and

estimating the nearest physical location and the farthest physical location of the tip according to the lower bound and the upper bound of the object-camera distance, respectively, and further according to the direction vector.

3. The method of claim 1, wherein the determining of the 3D location of the tip is performed only after a reference surface is detected absent within the FOV.

4. The method of claim 1, wherein the structured-light pattern is reconfigurable by the light source and is adapted according to the nearest and farthest physical locations that are estimated.

5. The method of claim 1, wherein the structured-light pattern is a static pattern invariant to the nearest and farthest physical locations calculated in the determining of the 3D location of the tip.

6. The method of claim 5, wherein the static pattern is periodic in intensity and has a pre-determined wavelength for enabling unique determination of the 3D location of the tip that is located inside a pre-defined space.

7. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 1.

8. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 2.

9. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 3.

10. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 4.

11. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 5.

12. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 6.

13. The method of claim 1, further comprising:

when both the finger-like object and a reference surface are detected present within the FOV, determining a height of the tip above the reference surface so that one or more of the user inputs are determinable according to the tip'"'"'s height;

wherein the determining of the tip'"'"'s height above the reference surface comprises;

determining parameters of a surface-plane equation for geometrically characterizing the reference surface;

determining a height of the camera and a height of the light source above the reference surface according to the parameters of the surface-plane equation;

projecting the plain sheet of light to an area that substantially covers at least a region-of-interest (ROI) comprising an area surrounding and including the tip, such that the object around the tip is illuminated to form a shadow on the reference surface unless the object is substantially close to the reference surface;

estimating a camera-observed shadow length from a ROI-highlighted image, wherein the ROI-highlighted image is captured by the camera after the plain sheet of light is projected, and wherein the camera-observed shadow length is a length of a part of the shadow formed on the reference surface along a topographical surface line and observed by the camera;

estimating a shadow-light source distance from the ROI-highlighted image; and

estimating the tip'"'"'s height above the reference surface based on a set of data including the surface-plane equation, the camera-observed shadow length, the shadow-light source distance, a distance measured in a direction parallel to the reference surface between the light source and the camera, the height of the camera above the reference surface, and the height of the light source above the reference surface.

14. The method of claim 13, further comprising:

determining that the object touches the reference surface if the tip'"'"'s height above the reference surface is less than a pre-determined threshold; and

determining one or more of the user inputs according to whether the object touches the reference surface.

15. The method of claim 13, further comprising:

when the object is detected within the FOV, detecting whether the reference surface is also present within the FOV;

wherein the detecting of whether the reference surface is also present within the FOV comprises;

in an absence of the light source illuminating the object, capturing the FOV by the camera to give a first image;

in a presence of the light source illuminating the object with the plain sheet of light, capturing a second image containing the object by the camera;

from each of the first and second images, determining an intensity level of a surrounding area outside and adjacent a boundary of the object; and

determining that the reference surface is present if the intensity level determined from the first image is substantially different from the intensity level determined from the second image.

16. The method of claim 13, wherein the estimating of the tip'"'"'s height above the reference surface comprises computing

17. The method of claim 13, further comprising:

obtaining a surface profile of the reference surface, and a surface map configured to map any point on an image captured by the camera to a corresponding physical location on the reference surface;

wherein;

the camera-observed shadow length and the shadow-light source distance are estimated from the ROI-highlighted image by using the surface map; and

the set of data further includes the surface profile.

18. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 13.

19. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 14.

20. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 15.

21. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 16.

22. A wearable device comprising one or more processors, a camera and a light source, the light source being configured to generate a plain sheet of light and a structured-light pattern, the camera having an image sensor and having an optical center located at a distance of a focal length from the image sensor, wherein the wearable device is configured to execute a process for receiving user inputs according to the method of claim 17.