Method and apparatus for applying free space input for surface constrained control

• US 10,401,966 B2
• Filed: 05/15/2015
• Issued: 09/03/2019
• Est. Priority Date: 05/15/2015
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Kandadai, Shashwat, Abercrombie, Nathan, Chen, Yu-Hsiang, Itani, Sleiman
• Primary Examiner(s): Azari, Sepehr
• Application Number:
  US14/713,954
• Publication Number:
  US 20160334875A1
• Time in Patent Office: 1,572 Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A free space input standard is instantiated on a processor. Free space input is sensed and communicated to the processor. If the free space input satisfies the free space input standard, a touch screen input response is invoked in an operating system. The free space input may be sensed using continuous implicit, discrete implicit, active explicit, or passive explicit approaches. The touch screen input response may be invoked through communicating virtual touch screen input, a virtual input event, or a virtual command to or within the operating system. In this manner free space gestures may control existing touch screen interfaces and devices, without modifying those interfaces and devices directly to accept free space gestures.

Claims

1. A method, comprising:

determining, by a processor, a surface constrained input standard for a surface constrained input, wherein the surface constrained input is an input generated approximate to a physical surface along a two dimensional physical plane;

sensing, by a sensor, the surface constrained input generated in a three-dimensional space approximate to the physical surface by a first digit of a hand of an individual, wherein the sensor is integrated into a head-mounted device and the surface constrained input is sensed by the sensor relative to a head of the individual;

defining, by the processor, a height, a width, and a depth of a volumetric three dimensional virtual boundary approximate and removed from the physical surface based on the surface constrained input, wherein the height, the width and the depth of the volumetric three dimensional virtual boundary is defined by the three-dimensional space where the surface constrained input is received in relation to the physical surface as the first digit moves toward the physical surface, interacts with the physical surface, and moves away from the physical surface;

determining, by the processor, a free space input standard for a free space input, the tree space input being received within an augmented reality construct defined by the same approximate height, width, and depth of the volumetric three dimensional virtual boundary wherein;

a portion of the volumetric three dimensional virtual boundary abuts a virtual object in the augmented reality construct;

associating, by the processor, the free space input with the surface constrained input;

sensing, by a sensor, a first movement of the first digit of the hand of the individual within the volumetric space, wherein the first movement comprises the first digit moving from a first plane in the volumetric space to a second plane in the volumetric space;

sensing, by the sensor, a second movement of the first digit within the volumetric space, wherein the second movement comprises the first digit moving horizontally along the second plane;

sensing, by the sensor, a third movement of the first digit within the volumetric space, wherein the third movement comprises the first digit moving from the first plane In the volumetric space to a third plane in the volumetric space, wherein the first movement, the second movement, and the third movement mimic the surface constrained input within the volumetric space;

sensing, by the sensor, a fourth movement of a second digit of the hand within the volumetric space;

determining that the free space input standard includes the first movement of the first digit, the second movement of the first digit, and the third movement of the first digit, and excludes the fourth movement of the second digit;

generating the free space input that includes the first movement of the first digit, the second movement of the first digit, and the third movement of the first digit, and excludes the fourth movement of the second digit;

determining, by the processor, that the free space input satisfies the free space input standard;

in response to the free space input satisfying the free space input standard, generating a virtual surface constrained input that satisfies the surface constrained input standard associated with the surface constrained input; and

executing, by the processor, a surface constrained input response, wherein the surface constrained input response associated with the virtual surface constrained input is the same response as a response executed for the surface constrained input.

2. The method of claim 1, wherein the free space input standard comprises a three dimensional input standard.

3. The method of claim 1, wherein the free space input standard comprises at least one of a gesture standard or a posture standard.

4. The method of claim 1, wherein the free space input standard comprises a standard for motion of an end effector in free space.

5. The method of claim 1, wherein the surface constrained input comprises a touch screen input.

6. The method of claim 1, wherein the surface constrained input comprises a touchdown input.

7. The method of claim 1, wherein the surface constrained input comprises a touchdown and swipe input.

8. The method of claim 1, wherein sensing the free space input comprises at least one of imaging, stereo imaging, video imaging, or depth imaging the free space input.

9. The method of claim 1, wherein sensing the free space input comprises an evaluation over time.

10. The method of claim 9, wherein sensing the free space input comprises evaluating a plurality of images at intervals over time.

11. The method of claim 1, further comprising controlling an apparatus with the surface constrained input response.

12. The method of claim 1, wherein the free space input standard comprises a plurality of end effector standards for at least one end effector, considered collectively therefor with regard to said free space input.

13. The method of claim 12, wherein the free space input standard comprises at least one of a standard for speed, velocity, position, orientation, direction, acceleration, joint angle, separation, extension, or selection of the at least one end effector.

14. The method of claim 12, wherein the free space input standard comprises a profile of a finger-extended side to side swipe considering path shape, velocity, acceleration, or joint position over time.

15. The method of claim 14, wherein the free space input standard comprises a standard for at least one of speed, velocity, position, orientation, direction, acceleration, joint angle, separation, extension, or selection of the at least one end effector.

16. The method of claim 1, wherein the free space input standard comprises a plurality of end effector motion standards for at least one end effector considered sequentially with regard to a first sub-standard and a second sub-standard, the first sub-standard corresponding to a first portion of the surface constrained input and the second sub-standard corresponding to a second portion of the surface constrained input, the first and second portions considered sequentially with regard to the free space input.

17. The method of claim 1, wherein the free space input standard comprises a plurality of end effector motion standards for at least one end effector and an indication at least substantially contemporary therewith.

18. The method of claim 17, wherein the indication comprises at least one of a hand posture, a hand gesture, an eye posture, an eye gesture, or a voice command.

19. The method of claim 17, wherein the free space input standard comprises a standard for at least one of speed, velocity, position, orientation, direction, acceleration, joint angle, separation, extension, or selection of said at least one end effector.

20. The method of claim 1, wherein the free space input standard comprises at least one of a plurality of state standards.

21. The method of claim 20, wherein the plurality of state standards are adapted to be combined to form a plurality of free space input standards, each free space input standard corresponding with the free space input.

22. The method of claim 20, wherein each of said at least one of said state standards comprises a standard for a motion of an end effector.

23. The method of claim 1, wherein generating the virtual surface constrained input comprises deriving the virtual surface constrained input from the free space input.

24. The method of claim 1, wherein generating the virtual surface constrained input comprises reproducing a fixed virtual surface constrained input.

25. The method of claim 1, wherein the free space input standard excludes one or more movements included in the surface constrained input standard.

26. An apparatus, comprising:

means for determining a surface constrained input standard for a surface constrained input, wherein the surface constrained input is a first input generated approximate to a physical surface along a two dimensional physical plane;

means for sensing the surface constrained input generated in a three-dimensional space approximate to the physical surface by a first digit of a hand of an Individual wherein the means for sensing is integrated into a head-mounted device and the surface constrained input is sensed by the means for sensing relative to a head of the individual;

means for defining a height, a width, and a depth of a volumetric three dimensional virtual boundary approximate and removed from the physical surface based on the surface constrained input wherein the height, the width and the depth of the volumetric three dimensional virtual boundary is defined by the three-dimensional space where the surface constrained input is received in relation to the physical surface as the first digit moves toward the physical surface, interacts with the physical surface, and moves away from the physical surface;

means for determining a free space input standard for a free space input, the free space input is a second input generated within an augmented reality construct defined by the same approximate height, width, and depth of the volumetric three dimensional virtual boundary that is generated in an augmented reality and is bounded by a virtual surface, wherein;

a portion of the volumetric three dimensional virtual boundary abuts a virtual object in the augmented reality construct;

means for associating the free space input with the surface constrained input;

means for sensing a first movement of the first digit of the hand of the individual within the volumetric space, wherein the first movement comprises the first digit moving from a first plane in the volumetric space to a second plane in the volumetric space;

means for sensing a second movement of the first digit within the volumetric space, wherein the second movement comprises the first digit moving horizontally along the second plane;

means for sensing a third movement of the first digit within the volumetric space, wherein the third movement comprises the first digit moving from the first plane in the volumetric space to a third plane in the volumetric space, wherein the first movement, the second movement, and the third movement mimic the surface constrained input within the volumetric space;

means for sensing a fourth movement of a second digit of the hand within the volumetric space;

means for determining that the free space input standard includes the first movement of the first digit, the second movement of the first digit; and

the third movement of the first digit, and excludes the second movement of the second digit;

means for generating the free space input that includes the first movement of the first digit, the second movement of the first digit, and the third movement of the first digit, and excludes the fourth movement of the second digit;

means for determining whether the free space input satisfies the free space input standard; and

in response to the free space input satisfying the free space input standard, means for generating a virtual surface constrained input that satisfies the surface constrained input standard associated with the surface constrained input; and

means for executing a surface constrained input response, wherein the surface constrained input response associated with the virtual surface constrained input is the same response as a response executed for the surface constrained input.

27. An apparatus, comprising:

a sensor adapted to;

sense a first movement of a first digit of a hand of an individual within a volumetric space, wherein;

the first movement comprises the first digit moving from a first plane in the volumetric space to a second plane in the volumetric space; and

sense a second movement of the first digit within the volumetric space, wherein the second movement comprises the first digit moving horizontally along the second plane;

sense a third movement of the first digit within the volumetric space, wherein the third movement comprises the first digit moving from the first plane in the volumetric space to a third plane in the volumetric space, wherein the first movement, the second movement, and the third movement mimic the surface constrained input within the volumetric space;

sense a fourth movement of a second digit of the hand within the volumetric space;

sense a surface constrained input generated in a three-dimensional space approximate to the physical surface by the first digit of the hand of the individual wherein the sensor is integrated into a head-mounted device and the surface constrained input is sensed by the sensor relative to a head of the individual;

a processor coupled to the sensor, wherein the processor is to;

determine, a surface constrained input standard for the surface constrained input, wherein the surface constrained input is an input generated approximate to the physical surface along a two dimensional physical plane;

define a height, a width, and a depth of a volumetric three dimensional virtual boundary approximate the physical surface based on the surface constrained input wherein the height, the width and the depth of the volumetric three dimensional virtual boundary is defined by the three-dimensional space where the surface constrained input is received in relation to the physical surface as the first digit moves toward the physical surface, interacts with the physical surface, and moves away from the physical surface;

determine a free space input standard for a free space input, the free space input being received within an augmented reality construct defined by the same approximate height, width, and depth of the volumetric three dimensional virtual boundary, wherein;

the volumetric three dimensional virtual boundary comprises a volumetric space removed from the physical surface that includes a bounded three dimensional space with a height, a width, and a depth; and

a portion of the volumetric three dimensional virtual boundary abuts a virtual object in the augmented reality construct;

determine that the free space input standard includes the first movement of the first digit, the second movement of the first digit, and the third movement of the first digit, and excludes the fourth movement of the second digit;

generate the free space input that includes the first movement of the first digit, the second movement of the first digit, and the third movement of the first digit, and excludes the fourth movement of the second digit;

generate a virtual surface constrained input that satisfies the surface constrained input standard; and

execute a surface constrained input response, wherein the surface constrained input response associated with the virtual surface constrained input is the same response as a response executed for the surface constrained input.

28. The apparatus of claim 27, wherein the processor and the sensor are disposed on a head mounted display.