Gesture detection using an array of short-range communication devices

• US 9,870,057 B1
• Filed: 01/27/2014
• Issued: 01/16/2018
• Est. Priority Date: 02/02/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.), New Jersey Institute of Technology
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Want, Roy, Li, Yang, Schilit, William Noah
• Primary Examiner(s): Pan, Yuwen
• Assistant Examiner(s): Chen, Zhitong
• Application Number:
  US14/165,167
• Time in Patent Office: 1,450 Days
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/046 by electromagnetic means
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

In general, techniques and systems for defining a gesture with a computing device using short-range communication are described. In one example, a method includes obtaining position information from an array of position devices using near-field communication (NFC) during a movement of the computing device with respect to the array, wherein the position information identifies unique positions within the array for each position device from which position information was obtained. The method may also include determining sequence information associated with the position information, wherein the sequence information is representative of an order in which the position information was obtained from each position device, and performing, by the computing device, an action based at least in part on the position information and the sequence information, wherein the position information and the sequence information are representative of a gesture input associated with the movement of the computing device.

Claims

1. A method comprising:

receiving, by a computing device, an initiation input indicative of a command to obtain position information;

during movement of the computing device with respect to an array of position devices and while receiving the initiation input, obtaining, by the computing device and from a near-field communication module operatively coupled to the computing device, the position information from the array using near-field communication, wherein the position information identifies unique positions of respective position devices within the array;

determining, by the computing device, sequence information associated with the position information, wherein the sequence information is representative of an order in which the position information was obtained from each position device within the array;

responsive to obtaining the position information from a first position device of the array of position devices, determining, by the computing device and based on input from a sensor different than the near-field communication module, the movement of the computing device during a time period that includes a time when the position information was obtained from the array of position devices; and

performing, by the computing device, an action based at least in part on each of the position information, the sequence information, and the determined movement, wherein the position information, the sequence information, and the determined movement are representative of a gesture input.

2. The method of claim 1, further comprising:

refining, based at least in part on the determined movement, at least one of a shape or a length of the gesture input represented by the position information and the sequence information.

3. The method of claim 1, further comprising:

transmitting, by the computing device, the position information, the sequence information, and the determined movement to a networked device via a network; and

receiving, by the computing device, gesture information from the networked device based at least in part on the position information, the sequence information, and the determined movement, wherein the gesture information comprises a command to perform the action.

4. The method of claim 3, further comprising:

generating at least one uniform resource locator representative of the position information from at least one of the position devices in the array and at least a portion of the sequence information.

5. The method of claim 1, wherein determining the sequence information comprises adding a time stamp to the position information that indicates when each portion of the position information was obtained from each respective position device.

6. The method of claim 1, further comprising:

capturing, by the computing device, an image of a printed graphic associated with the array; and

selecting, by the computing device, the action based at least in part on the gesture input and the captured image.

7. The method of claim 1, further comprising:

presenting an image on a display operatively coupled to the computing device prior to obtaining the position information; and

selecting, by the computing device, the action based at least in part on the gesture input and the image.

8. The method of claim 1, further comprising:

responsive to no longer receiving the initiation input, terminating the obtaining of the position information.

9. The method of claim 1, wherein:

receiving the initiation input comprises receiving a first initiation input and a second initiation input;

obtaining the position information comprises;

obtaining a first set of position information from a first subset of the array of position devices in response to receiving the first initiation input; and

obtaining a second set of position information from a second subset of the array of position devices in response to receiving the second initiation input, and

the method further comprising;

defining the gesture input based on the first set of position information and the second set of position information.

10. The method of claim 1, wherein the gesture input comprises one of a question mark, a plus sign, a minus sign, a checkmark, a number, or a letter.

11. The method of claim 1, wherein the computing device is a mobile device.

12. The method of claim 1, wherein determining the movement of the computing device comprises determining, based on the input from the sensor, the movement of the computing device between a first known position and a second known position, wherein each of the first known position and the second known position correspond to a position of a respective position device of the array of position devices.

13. The method of claim 1, wherein obtaining the position information from the array comprises obtaining the position information from a first position device of the array, the method further comprising:

responsive to obtaining the position information from the first position device, changing a detection rate of the sensor from a first detection rate to a second detection rate.

14. The method of claim 1, wherein obtaining the position information comprises determining, by the computing device and based at least in part on a signal strength of the position information, a location of the computing device.

15. A non-transitory computer-readable storage medium encoded with instructions that cause one or more processors of a computing device to perform operations comprising:

receive, from an input device of the computing device, an indication of an initiation input that indicates a command to obtain position information;

during movement of the computing device with respect to an array of position devices and while the input device receives the initiation input, obtaining, from a near-field communication module operatively coupled to the computing device, the position information from the array using near-field communication, wherein the position information identifies unique positions of respective position devices within the array;

determining sequence information associated with the position information, wherein the sequence information is representative of an order in which the position information was obtained from each position device within the array;

responsive to obtaining the position information from a first position device of the array of position devices, determining, based on input from a sensor different than the near-field communication module, the movement of the computing device during a time period that includes a time when the position information was obtained from the array of position devices; and

performing an action based at least in part on each of the position information, the sequence information, and the determined movement, wherein the position information, the sequence information, and the determined movement are representative of a gesture input.

16. The non-transitory computer-readable storage medium of claim 15, further comprising instructions that cause one or more processors of the computing device to perform operations comprising:

refining, based at least in part on the determined movement, at least one of a shape or a length of the gesture input represented by the position information and the sequence information.

17. The non-transitory computer-readable storage medium of claim 15, further comprising instructions that cause one or more processors of the computing device to perform operations comprising:

transmitting the position information, the sequence information, and the determined movement to a networked device via a network; and

receiving gesture information from the networked device based at least in part on the position information, the sequence information, and the determined movement, wherein the gesture information comprises a command to perform the action.

18. The non-transitory computer-readable storage medium of claim 15, further comprising instructions that cause one or more processors of the computing device to perform operations comprising:

responsive to the input device no longer receiving the initiation input, terminating the obtaining of the position information.

19. The non-transitory computer-readable storage medium of claim 15, wherein:

the instructions that cause one or more processors of the computing device to perform operations comprising receiving the initiation input comprises instructions that cause one or more processors of the computing device to perform operations comprising receiving a first initiation input and a second initiation input;

the instructions that cause one or more processors of the computing device to perform operations comprising obtaining the position information comprises instructions that cause one or more processors of the computing device to perform operations comprising;

obtaining a first set of position information from a first subset of the array of position devices in response to receiving the first initiation input; and

obtaining a second set of position information from a second subset of the array of position devices in response to receiving the second initiation input, and the non-transitory computer-readable storage medium further comprising;

instructions that cause one or more processors of the computing device to perform operations comprising defining the gesture input based on the first set of position information and the second set of position information.

20. The non-transitory computer-readable storage medium of claim 15, wherein the gesture input comprises one of a question mark, a plus sign, a minus sign, a checkmark, a number, or a letter.

21. The non-transitory computer-readable storage medium of claim 15, wherein the computing device is a mobile device.

22. A mobile computing device comprising:

an input device configured to receive an initiation input indicative of a command to obtain position information;

a near-field communication module configured to, during movement of the mobile computing device with respect to an array of position devices and while the input device receives the initiation input, obtain the position information from the array using near-field communication, wherein the position information identifies unique positions of respective position devices within the array;

one or more processors configured to determine sequence information associated with the position information, wherein the sequence information is representative of an order in which the position information was obtained from each position device within the array; and

a sensor configured to determine the movement of the mobile computing device during a time period that includes a time when the position information was obtained from the array of position devices, wherein the sensor is different than the near-field communication module, and wherein the sensor is configured to determine the movement of the mobile computing device in response to obtaining the position information from a first position device of the array of position devices,

wherein the one or more processors are configured to perform an action based at least in part on each of the position information, the sequence information, and the determined movement, wherein the position information, the sequence information, and the determined movement are representative of a gesture input.

23. The mobile computing device of claim 22, wherein the near-field communication module is further configured to terminate the obtaining of the position information in response to the input device no longer receiving the initiation input.