Gesture detection and interactions

• US 10,203,763 B1
• Filed: 10/05/2015
• Issued: 02/12/2019
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Poupyrev, Ivan, Schwesig, Carsten, Schulze, Jack, Arnall, Timo, Bishop, Durrell Grant Bevington
• Primary Examiner(s): Pardo, Thuy
• Application Number:
  US14/874,955
• Time in Patent Office: 1,226 Days
• Field of Search: 345173, 345156, 345426, 345634, 345174
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Gesture detection and interaction techniques are described. Object detection used to support the gestures may be accomplished in a variety of ways, such as by using radio waves as part of a radar technique. In a first example, the techniques are implemented such that one hand of a user sets a context for a gesture that is defined by another hand of the user. In another example, a gesture recognition mode is utilized. In yet another example, detection of distance is used such that the same motions may be used to different between operations performed. In a further example, split gestures are supported. In another instance, entry into a gesture recognition mode may be implemented through touch and then recognized through three-dimensional orientation and motion of that hand or another.

Claims

1. A computing device comprising:

a housing;

a three dimensional object detection system disposed within the housing and configured to detect hands of a user in three dimensional space by utilizing a signal of a sufficient strength to pass through an intervening article worn by or associated with the user; and

a gesture module implemented at least partially in hardware and disposed within the housing, the gesture module configured to cause performance of one or more operations by the computing device responsive to identification of gestures from inputs involving detection of the hands of the user in three dimensional space by the three dimensional object detection system through the intervening article of clothing of the user, the gestures being identified without receiving a touch-input to the computing device, a first gesture performed by a first hand of the user defining the one or more operations and a second gesture performed by a second hand of the user defining a scale for the one or more operations.

2. The computing device of claim 1, wherein the housing is configured to the held or worn by the user.

3. The computing device of claim 1, wherein the three dimensional object detection system is configured to perform the detection using radar techniques involving radio waves.

4. The computing device of claim 3, wherein the radio waves correspond to a frequency band included as part of a Wi-Fi radio spectrum.

5. The computing device of claim 1, wherein the three dimensional object detection system is configured to identify the hands of the user and detect an orientation or movement of the hands of the user.

6. The computing device of claim 1, wherein the first and second gestures involve fingers of the hands of the user.

7. The computing device of claim 1, wherein the three dimensional object detection system is configured to detect the object through a fabric or plastic material.

8. A method of controlling operation of a computing device based on gesture detection, the method comprising:

detecting a first gesture performed by a first hand of a user using a three dimensional object detection system of the computing device that is able to determine gestures through an intervening article of clothing of the user without a touchscreen input, the first gesture defining an operation to be performed by the computing device;

detecting a second gesture performed by a second hand of the user using the three dimensional object detection system, the second gesture defining a scale for the operation to be performed by the computing device; and

controlling performance of the operation by the computing device based on the first and second inputs.

9. The method of claim 8, wherein the computing device is configured to be held or worn by the user.

10. The method of claim 8, wherein the three dimensional object detection system is configured to perform the detection using radar techniques involving radio waves.

11. The method of claim 10, wherein the radio waves correspond to a frequency band included as part of a Wi-Fi radio spectrum.

12. The method of claim 8, wherein the second gesture comprises raising or lowering the second hand.

13. The method of claim 8, wherein the operation comprises a forward or backward navigation through a user interface, a scrolling, a zooming, a change in volume, or a change in channel.

14. One or more non-transitory computer-readable storage media devices containing instructions that when executed by one or more processors causes the processors to implement operations comprising:

receiving a first indication of a first gesture performed by a first hand of a user without receiving a touchscreen input, the indication received from a three dimensional object detection system that utilizes a signal of a sufficient strength to pass through an intervening article worn by or associated with the user;

receiving, from the three dimensional object detection system, an indication of a second gesture performed by a second hand of the user without receiving a touchscreen input;

performing an operation based on the first and second indications, the first indication of the first gesture defining the operation, and the second indication corresponding to the second gesture defining a scale for the operation.

15. The one or more non-transitory computer-readable storage media devices of claim 14, wherein the signal comprises a radio wave.

16. The one or more non-transitory computer-readable storage media devices of claim 15, wherein the radio wave corresponds to a frequency band included as part of a wireless local area network band.

17. The one or more non-transitory computer-readable storage media devices of claim 14, wherein the first and second indications comprise orientations or movements of the first and second hands, respectively.

18. The one or more non-transitory computer-readable storage media devices of claim 14, wherein the second gesture comprises raising or lowering the first hand.

19. The one or more non-transitory computer-readable storage media devices of claim 14, wherein the first gesture comprises a twisting motion.

20. The one or more non-transitory computer-readable storage media devices of claim 14, wherein the operation comprises a forward or backward navigation through a user interface, a scrolling, a zooming, a change in volume, or a change in channel.