Detection of an input object on or near a surface

• US 10,289,203 B1
• Filed: 03/04/2013
• Issued: 05/14/2019
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Liu, Yue, Yao, Ning
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Nokham, James S
• Application Number:
  US13/783,854
• Time in Patent Office: 2,262 Days
• Field of Search: 345173-175, 345418-419, 345660, 348 46- 48, 715702-705
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

A number of images of an environment may be obtained over time by a number of sensors. The images may include an input surface and an input object, such as a hand or stylus, that may be utilized by a user to provide an input via the input surface. In some cases, the input may be provided without the input object touching the input surface. The location of the input object with respect to the input surface over time and a speed of the input object over time may be used to determine whether an input has been indicated by the input object without contacting the input surface.

Claims

1. A system comprising:

one or more processors;

one or more cameras to obtain a plurality of images of an environment, the one or more cameras including a 2-dimensional (2D) camera, a 3-dimensional (3D) camera, or both, and the environment including a finger and an input surface, wherein a user interface is displayed via at least a portion of the input surface;

one or more projectors to project an image onto the input surface in the environment, at least one of the one or more cameras is separated from at least one of the one or more projectors by a linear offset; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

determining a first location of the finger in the environment based, at least in part, on first information associated with a first image of the plurality of images;

determining a first distance between the first location and a portion of an exterior face of the input surface;

determining a second location of the finger in the environment based, at least in part, on second information associated with a second image of the plurality of images, the second image being obtained subsequent to the first image;

determining a second distance between the second location and the portion of the exterior face;

determining that the first distance and the second distance are less than a first threshold distance;

determining that the second distance is less than the first distance;

determining that a touch event has occurred based, at least in part, on;

the first distance and the second distance being less than the first threshold distance,

the second distance being less than the first distance,

a difference between the first distance and the second distance being less than a second threshold distance, wherein the difference is associated with movement from the first location to the second location; and

a speed associated with the movement estimated based, at least in part, on the difference and an amount of time between obtaining the first image and the second image;

determining a location on the input surface associated with the touch event based at least in part on the linear offset; and

identifying an input based, at least in part, on the location associated with the touch event.

2. The system of claim 1, wherein determining that the touch event has occurred is further based on the speed being less than a threshold speed.

3. The system of claim 1, wherein the acts further comprise:

determining that the finger contacts the input surface; and

determining that a surface touch event has occurred based at least in part on the finger contacting the input surface.

4. The system of claim 1, wherein the acts further comprise detecting a shadow on the input surface, and wherein determining the location on the input surface indicated by the finger is based at least in part on detecting the shadow.

5. A method comprising:

obtaining a plurality of images of an environment;

identifying an input object and an input surface in the environment;

determining a first distance between a first location of the input object and a location of the input surface in the environment based, at least in part, on information associated with a first image of the plurality of images;

determining a second distance between a second location of the input object and the location of the input surface in the environment based, at least in part, on information associated with a second image of the plurality of images, the second image being captured subsequent to the first image;

determining that the first distance and the second distance are no greater than a threshold distance from the input surface;

determining that a difference between the first distance and the second distance is less than a specified distance that is no greater than the threshold distance, wherein the difference represents movement of the input object from the first location to the second location;

determining a speed associated with the input object moving from the first location to the second location, wherein determining the speed is based at least in part on the plurality of images;

determining that the input object has stopped based on the speed being below a threshold speed; and

determining that a near surface touch event has occurred based, at least in part on, the difference between the first distance and the second distance being less than the specified distance and the determining that the input object has stopped.

6. The method of claim 5, wherein the input object includes a finger, a hand, a stylus, a pointing device, or a combination thereof.

7. The method of claim 5, wherein the plurality of images are captured via a 2-dimensional (2D) camera, a 3-dimensional (3D) camera, a red-green-blue (RGB) camera, or a combination thereof.

8. The method of claim 5, wherein a first portion of the plurality of images is obtained substantially at a first time and a second portion of the plurality of images is obtained substantially at a second time.

9. The method of claim 5, further comprising generating a response to the near surface touch event, wherein the response includes modifying content of a user interface displayed with respect to the input surface, displaying particular content via the input surface, or both.

10. The method of claim 5, wherein the input surface is touch sensitive.

11. The method of claim 5, wherein the input surface is included in an electronic device.

12. The method of claim 5, further comprising detecting a dragging input based at least in part on a first direction of motion of the input object being substantially perpendicular to the input surface and a second direction of motion of the input object being substantially parallel to the input surface.

13. One or more computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

obtaining a series of images of an environment, the series of images including an input object and an input surface in the environment;

causing a projector to project light onto the input surface;

determining a distance between a location of the input object and an exterior face of the input surface based at least in part on the series of images;

determining a speed of the input object moving towards the input surface based at least in part on a first location of the input object and a second location of the input object shown in the series of images; and

estimating a location on the input surface indicated by the input object based at least partly on determining that the distance is no greater than a threshold distance and that the speed is no greater than a threshold speed, wherein the threshold distance is determined at least in part based on the speed.

14. The one or more computer-readable media of claim 13, wherein the acts further comprise:

estimating the location on the input surface indicated by the input object using a plurality of techniques, wherein a respective weighting is assigned to a technique of the plurality of techniques.

15. The one or more computer-readable media of claim 14, wherein the respective weighting assigned to the technique is based at least in part on the distance, the speed, or both.

16. The one or more computer-readable media of claim 13, wherein the estimating the location of the input surface indicated by the input object includes:

identifying a shadow formed on the input surface by the input object and the light projected into the environment; and

identifying a portion of the shadow corresponding to a pointing portion of the input object; and

wherein the estimating the location includes determining a particular location on the input surface that includes the portion of the shadow corresponding to the pointing portion of the input object.

17. The one or more computer-readable media of claim 16, wherein the shadow is identified based, at least in part, on a contrast in brightness, color, texture, or a combination thereof, of a first portion of the input surface and a second portion of the input surface.

18. The one or more computer-readable media of claim 13, wherein the estimating the location includes extrapolating the location along a particular direction.

19. The one or more computer-readable media of claim 18, wherein the particular direction includes a direction that the input object is pointing, a direction that a user holding the input object is facing, or a direction of a projection axis of the projector.

20. The one or more computer-readable media of claim 13, wherein the projector projects a user interface onto the input surface and the location corresponds to a particular input feature of the user interface.

21. A system comprising:

one or more processors;

one or more cameras to obtain a plurality of images of an environment, the plurality of images captured in series over time and the environment including an input object and an input surface; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

determining a location of the input object and a location of the input surface based, at least in part, on analysis of one or more of the plurality of images;

determining that a near surface touch event has occurred based at least in part on a distance between the input object and the input surface being less than a specified distance based, at least in part, on a particular location of the input object and a particular location of the input surface in an image of the plurality of images;

determining that the near surface touch event has occurred based, at least in part, on a respective location of the input object in the plurality of images with respect to a threshold region and a direction of motion of the input object in the plurality of images, the direction of motion determined based, at least in part, on a change in position of the input object in individual ones of the plurality of images with respect to an order of the series and changes in time;

estimating a location on the input surface indicated by the input object based at least in part on determining that the distance between the location of the input object and an exterior face of the input surface is no greater than a threshold distance; and

providing a second indication indicating that the near surface touch event has ended based at least in part on determining that the input object is outside of the threshold region or determining that a surface touch event occurs.

22. The system of claim 21, wherein the acts further comprise determining a direction of motion of the input object based, at least in part, on a first location of the input object in the environment based, at least in part, on information associated with a first image of the plurality of images captured at a first time and a second location of the input object in the environment based, at least in part, on information associated with a second image of the plurality of images captured at a second time, the second time being subsequent to the first time.

23. The system of claim 22, wherein the direction of motion of the input object is towards the input surface when the first location of the input object is farther away from the location of the input surface than the second location of the input object.

24. The system of claim 23, wherein the determining that the near surface touch event occurs is based at least in part on the direction of motion of the input object being towards the input surface.

25. The system of claim 22, wherein the direction of motion of the input object is away from the input surface based at least in part on the first location of the input object being closer to the location of the input surface than the second location of the input object.

26. The system of claim 22, wherein the determining that the near surface touch event has occurred includes identifying a change in the direction of motion of the input object.

27. The system of claim 22, wherein a dragging input is detected based at least in part on:

determining a first direction of motion of the input object being substantially perpendicular to the input surface;

determining that the near touch event has occurred; and

determining a second of motion of the input object being substantially parallel to the input surface.

28. The system of claim 21, wherein the acts further comprise determining a speed of the input object based at least in part on a difference between a first location of the input object in a first image of the plurality of images and a second location of the input object in a second image of the plurality of images and an amount of time between obtaining the first image and the second image of the plurality of images.

29. The system of claim 28, wherein determining that the near surface touch event has occurred is further based at least in part on the speed of the input object being less than a threshold speed.