Gesture recognition cloud command platform, system, method, and apparatus

• US 9,916,010 B2
• Filed: 05/18/2015
• Issued: 03/13/2018
• Est. Priority Date: 05/16/2014
• Status: Active Grant

• Current Assignee: Visa International Service Association (Visa, Inc.)
• Original Assignee: Visa International Service Association (Visa, Inc.)
• Inventors: Harris, Theodore, Edington, Scott, Faith, Patrick
• Primary Examiner(s): Okebato, Sahlu
• Application Number:
  US14/715,105
• Publication Number:
  US 20160109954A1
• Time in Patent Office: 1,030 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2358/00 Arrangements for display data security
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2380/04 Electronic labels
• CPC Class Codes: H04L 63/0861 : using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/68 : Gesture-dependent or behaviour-dependent

Abstract

Systems and methods described herein are for transmitting a command to a remote system. A processing system determines the identity of the user based on the unique identifier and the biometric information. Thereafter, a sensor detects a gesture performed by the user. The sensor is configured to detect the gesture performed by the user when the user is located within the detectable range of the wireless antenna. The processing system determines an action associated with the detected gesture based on the identity of the user and sends a command to a remote computer system to cause it to perform the action associated with the detected gesture.

Claims

1. A computer-implemented method for transmitting a command to a remote computer system, the computer-implemented method comprising:

detecting, via a first sensor, a unique identifier for a user device that is associated with a user, wherein the unique identifier is sent to the first sensor by the user device, the first sensor includes a wireless antenna, the detection of the unique identifier via the first sensor indicates that the user device is located within a detectable range of the wireless antenna, and both the user device and wireless antenna are located within a merchant store;

detecting, via a second sensor, biometric information for a plurality of customers located within the detectable range of the wireless antenna, wherein the second sensor includes a camera that is configured to detect the biometric information for each of the plurality of customers when the plurality of customers are located within the detectable range of the wireless antenna;

determining, using a processing system, an identity of the user based on the unique identifier and the detected biometric information, wherein at least a portion of the detected biometric information matches facial parameters of the user;

after the determining of the identity of the user, detecting, via the second sensor, a gesture performed by the user, wherein the gesture is directed to an item that is located within the merchant store, and the second sensor is configured to detect the gesture performed by the user when the user is located at one or more second locations within the detectable range of the wireless antenna;

identifying, using the processing system, an action associated with the detected gesture within a user-defined gesture table based on the identity of the user; and

sending, using the processing system, the command to the remote computer system, the command instructing the remote computer system to perform the action for the item, the action corresponding to the detected gesture from the user-defined gesture table.

2. The computer-implemented method of claim 1, further comprising:

providing the biometric information, the unique identifier, and the gesture to a computer system, the computer system being connected to the first sensor and the second sensor, wherein the computer system sends the command to the remote computer system.

3. The computer-implemented method of claim 1, wherein the detected biometric information includes an image of at least a portion of a face of the user, the computer-implemented method further comprising:

analyzing the image of the at least the portion of the face of the user using the processing system, wherein the analyzing of the image generates an output indicating a facial characteristic of the user; and

comparing the output with a set of previously stored reference signals, wherein the determining of the identity of the user is based on the comparison.

4. The computer-implemented method of claim 1, wherein the unique identifier includes a media access control (MAC) address of the user device.

5. The computer-implemented method of claim 1, further comprising:

detecting, via the first sensor, a second unique identifier for a second user device that is associated with a second user, the detection of the second unique identifier indicating that the second user device is located within the detectable range of the wireless antenna, wherein the second user is a person for whom biometric information is not detected via the second sensor;

transmitting a message to the second user device;

after the transmitting of the message to the second user device, determining one or more users within the detectable range of the wireless antenna that make a motion indicating a receipt of a message via a user device;

detecting second biometric information for the one or more users via the second sensor; and

determining the identity of the second user based on the second unique identifier and the second biometric information.

6. The computer-implemented method of claim 1, wherein the remote computer system comprises a cloud computing system, the cloud computing system being configured to perform the action associated with the detected gesture.

7. The computer-implemented method of claim 1, wherein the action associated with the detected gesture includes:

a posting of a message on a social media website;

an initiation of a financial transaction between the user and a third party;

a transmittal of information to the user;

or

an action performed by a robot.

8. The computer-implemented method of claim 1, wherein the user causes the action to be associated with the gesture by interacting with a website, the website including an interface that permits the associating of the action with the gesture.

9. The computer-implemented method of claim 1, further comprising:

determining a candidate identity for the user based on the unique identifier for the user device, the candidate identity being associated with a previously-stored reference signal; and

after the determining of the candidate identity for the user, comparing the detected biometric information for the user with the previously-stored reference signal, wherein the identity of the user is determined based on the comparison.

10. The computer-implemented method of claim 1, wherein the second sensor includes one or more cameras that collect a series of images depicting the gesture being performed by the user, the series of images being collected over a period of time, and wherein the computer-implemented method further comprises:

analyzing, using the processing system, the series of images to determine a trajectory of an object within one or more images of the series of images; and

comparing the trajectory of the object with one or more previously stored trajectories, each of the previously stored trajectories being associated with a gesture.

13. The computer-implemented system of claim 1, wherein the detected biometric information includes an image of at least a portion of a face of the user, the computer-implemented system further comprising instructions to:

analyze the image of the at least the portion of the face of the user using the processing system, wherein the analyzing of the image generates an output indicating a facial characteristic of the user; and

compare the output with a set of previously stored reference signals, wherein the determining of the identity of the user is based on the comparison.

11. A computer-implemented system for transmitting a command to a remote computer system, the computer-implemented system comprising:

a memory; and

one or more processors disposed in communication with the memory and configured to issue processing instructions stored in the memory to;

detect, via a first sensor, a unique identifier for a user device that is associated with a user, wherein the unique identifier is sent to the first sensor by the user device, the first sensor includes a wireless antenna, the detection of the unique identifier via the first sensor indicates that the user device is located within a detectable range of the wireless antenna, and both the user device and wireless antenna are located within a merchant store;

detect, via a second sensor, biometric information for a plurality of customers located within the detectable range of the wireless antenna, wherein the second sensor includes a camera that is configured to detect the biometric information for each of the plurality of customers when the plurality of customers are located within the detectable range of the wireless antenna;

determine, using a processing system, an identity of the user based on the unique identifier and the detected biometric information, wherein at least a portion of the detected biometric information matches facial parameters of the user;

after the determining of the identity of the user, detecting, via the second sensor, a gesture performed by the user, wherein the gesture is directed to an item that is located within the merchant store, and the second sensor is configured to detect the gesture performed by the user when the user is located at one or more second locations within the detectable range of the wireless antenna;

identify, using the processing system, an action associated with the detected gesture within a user-defined gesture table based on the identity of the user; and

send, using the processing system, the command to the remote computer system, the command instructing the remote computer system to perform the action for the item, the action corresponding to the detected gesture from the user-defined gesture table.

12. The computer-implemented system of claim 11, further comprising instruction to:

provide the biometric information, the unique identifier, and the gesture to a computer system, the computer system being connected to the first sensor and the second sensor, wherein the computer system sends the command to the remote computer system.

14. The computer-implemented system of claim 11, wherein the unique identifier includes a media access control (MAC) address of the user device.

15. The computer-implemented system of claim 11, further comprising instruction to:

detect, via the first sensor, a second unique identifier for a second user device that is associated with a second user, the detection of the second unique identifier indicating that the second user device is located within the detectable range of the wireless antenna, wherein the second user is a person for whom biometric information is not detected via the second sensor;

transmit a message to the second user device;

after the transmitting of the message to the second user device, determine one or more users within the detectable range of the wireless antenna that make a motion indicating a receipt of a message via a user device;

detect second biometric information for the one or more users via the second sensor; and

determine the identity of the second user based on the second unique identifier and the second biometric information.

16. The computer-implemented system of claim 11, wherein the remote computer system comprises a cloud computing system, the cloud computing system being configured to perform the action associated with the detected gesture.

17. The computer-implemented system of claim 11, wherein the action associated with the detected gesture includes:

a posting of a message on a social media website;

an initiation of a financial transaction between the user and a third party;

a transmittal of information to the user;

or

an action performed by a robot.

18. The computer-implemented system of claim 11, wherein the user causes the action to be associated with the gesture by interacting with a website, the website including an interface that permits the associating of the action with the gesture.

19. The computer-implemented system of claim 11, further comprising instructions to:

determine a candidate identity for the user based on the unique identifier for the user device, the candidate identity being associated with a previously-stored reference signal; and

after the determining of the candidate identity for the user, compare the detected biometric information for the user with the previously-stored reference signal, wherein the identity of the user is determined based on the comparison.

20. The computer-implemented system of claim 11, wherein the second sensor includes one or more cameras that collect a series of images depicting the gesture being performed by the user, the series of images being collected over a period of time, and wherein the computer-implemented system further comprises instructions to:

analyze, using the processing system, the series of images to determine a trajectory of an object within one or more images of the series of images; and

compare the trajectory of the object with one or more previously stored trajectories, each of the previously stored trajectories being associated with a gesture.

21. A processor-readable non-transitory medium storing processor-issuable instructions for transmitting a command to a remote computer system, said instructions being configured to:

detect, via a first sensor, a unique identifier for a user device that is associated with a user, wherein the unique identifier is sent to the first sensor by the user device, the first sensor includes a wireless antenna, the detection of the unique identifier via the first sensor indicates that the user device is located within a detectable range of the wireless antenna, and both the user device and wireless antenna are located within a merchant store;

detect, via a second sensor, biometric information for a plurality of customers located within the detectable range of the wireless antenna, wherein the second sensor includes a camera that is configured to detect the biometric information for each of the plurality of customers when the plurality of customers are within the detectable range of the wireless antenna;

determine, using a processing system, an identity of the user based on the unique identifier and the detected biometric information, wherein at least a portion of the detected biometric information matches facial parameters of the user;

after the determining of the identity of the user, detecting, via the second sensor, a gesture performed by the user, wherein the gesture is directed to an item that is located within the merchant store, and the second sensor is configured to detect the gesture performed by the user when the user is located at one or more second locations within the detectable range of the wireless antenna;

identify, using the processing system, an action associated with the detected gesture within a user-defined gesture table based on the identity of the user; and

send, using the processing system, the command to the remote computer system, the command instructing the remote computer system to perform the action for the item, the action corresponding to the detected gesture from the user-defined gesture table.