Identifying consumers in a transaction via facial recognition

• US 9,619,803 B2
• Filed: 04/30/2015
• Issued: 04/11/2017
• Est. Priority Date: 04/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chandrasekaran, Sashikanth, Kalenichenko, Dmitry, Zwiebel, Timothy Raymond
• Primary Examiner(s): Masud, Rokib
• Application Number:
  US14/701,517
• Publication Number:
  US 20160321671A1
• Time in Patent Office: 712 Days
• Field of Search: 705 11, 705 26, 705 21
• CPC Class Codes: G06Q 20/202 Interconnection or interaction of plural electronic cash registers [ECR] or to host computer, e.g. network details, transfer of information from host to ECR or from ECR to ECR
• CPC Class Codes: G06Q 20/206 comprising security or operator identification provisions, e.g. password entry
• CPC Class Codes: G06Q 20/3224 Transactions dependent on location of M-devices
• CPC Class Codes: G06Q 20/326 Payment applications installed on the mobile devices
• CPC Class Codes: G06Q 20/40145 Biometric identity checks
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A merchant and a user register with a payment processing system, which establishes a facial template based on a user image. The user signs into a payment application via a user computing device, which receives an identifier from a merchant beacon device to transmit to the payment processing system. The payment processing system transmits facial templates to the merchant camera device for other users who are also signed in to the payment application in range of the merchant beacon device. The merchant camera device compares a captured facial image against the received facial templates to identify the user. A merchant POS device operator selects an account of the user. The merchant POS device transmits transaction details to the payment processing system, which processes the transaction with an issuer system. The payment processing system receives an approval of the transaction authorization request and transmits a receipt to the merchant POS device.

Claims

1. A computer-implemented method to identify users at locations by comparing facial imaging of users against facial templates of users known to be at particular locations, comprising:

broadcasting, by a computing device, an identifier associated with the computing device at a location;

receiving, by the computing device and from one or more other computing devices, one or more facial templates, each facial template associated with a corresponding user associated with a corresponding user computing device that received the broadcasted identifier at the location and retransmitted the identifier to the one or more other computing devices, each facial template comprising a representation of a respective facial image of the respective user;

storing, by the computing device, the one or more received facial templates;

capturing, by a camera module of the computing device, a video feed of an environment external to the computing device;

extracting, by the computing device, a facial image of a particular user from the video feed;

generating, by the computing device, a facial template of the particular user based on the facial image of the particular user, wherein the facial template of the particular user is a representation of the facial image of the particular user;

determining, by the computing device, a similarity between the generated facial template of the particular user and each of the one or more stored facial templates;

identifying, by the computing device, a particular stored facial template as corresponding to the generated facial template of the particular user based on the determined similarity between the generated facial template of the particular user and the particular stored facial template exceeding a threshold value indicating that the stored facial template corresponds to the generated facial template of the particular user;

transmitting, by the computing device, a response to a request to identify a user comprising an indication of the identified particular user.

2. The method of claim 1, further comprising receiving, by the computing device, the request to identify a user.

3. The method of claim 1, wherein the environment external to the computing device comprises the particular user located in proximity to one or more merchant system point of sale devices associated with the computing device.

4. The method of claim 2, wherein the request to identify a user is received from a merchant system point of sale device associated with the computing device, and wherein the response to the request to identify a user is transmitted to the merchant point of sale device.

5. The method of claim 4, further comprising, processing, by the merchant point of sale device, a transaction using payment account information associated with the identified particular user.

6. The method of claim 3, further comprising:

determining, by the computing device and based on a frame of the video feed, a position of a face of the particular user and positions of each of the one or more point of sale devices at the location; and

assigning, by the computing device and based on distances between the face of the particular user and each of the one or more point of sale devices in the frame of the video feed, the particular user to a particular point of sale device of the one or more point of sale devices,

wherein the indication of the identity of the particular user further comprises an identifier of the particular point of sale device and an indication of the assignment of the particular user to the particular point of sale device.

7. A computer-implemented method to identify users at locations by comparing facial imaging of users against facial templates of users known to be at particular locations, comprising:

receiving, by one or more computing devices and from a particular user computing device, an account identifier of an account associated with a particular user associated with the particular user computing device and a beacon device identifier, wherein the particular user computing device retransmits the beacon device identifier received via a network from a beacon device at a location associated with the beacon device identifier;

retrieving, by the one or more computing devices, a facial template associated with the account of the particular user based on the account identifier of the account associated with the particular user;

identifying, by the one or more computing devices, the location associated with the beacon device identifier;

adding, by the one or more computing devices, the facial template associated with the account of the particular user and the associated location to a current customer log comprising facial templates corresponding to user computing devices that retransmit the beacon device identifier received from the beacon device at the location associated with the beacon device identifier;

transmitting, by the one or more computing devices and to a computing device at the location, the current customer log comprising the facial template associated with the account of the particular user, wherein the computing device identifies the particular user based on identifying the facial template associated with the account of the particular user from the current customer log as being similar to a facial template of the particular user generated from a facial image of the particular user captured by the computing device.

8. The method of claim 7, further comprising:

determining, by the one or more computing devices, that the particular user computing device is no longer maintains the network connection with the beacon device;

in response to determining that the particular user computing device no longer maintains the network connection with the beacon device, removing, by the one or more computing devices, the facial template associated with the account of the particular user and associated location from the current customer log to generate an updated current customer log; and

transmit, by the one or more computing devices and to the computing device, the updated current customer log.

9. The method of claim 8, wherein determining that the particular user computing device no longer maintains the network connection with the beacon device comprises detecting no subsequent transmission from the particular user computing device of the user account identifier and the beacon identifier within a threshold amount of time.

10. The method of claim 7, further comprising:

receiving, by the one or more computing devices and from the computing device, a facial template associated with an identified user, a transaction total, and a merchant account identifier from the computing device at the location;

determining, based on the received facial template associated with the identified user, an account identifier associated with the identified user;

transmitting, by the one or more computing devices and to a point of sale device, payment account information associated with a user account associated with the account identifier associated with the identified user;

receiving, by the one or more computing devices and from the point of sale device, an indication of a selection of particular payment account information from the payment account information for use in a transaction;

generating, by the one or more computing devices, a transaction authorization request comprising the transaction total, the merchant account identifier, and the particular payment account information item;

transmitting, by the one or more computing devices, the transaction authorization request to an issuer system associated with the particular payment account information;

receiving, by the one or more computing devices and from the issuer system, data comprising an authorization of the transaction authorization request; and

transmitting, by the one or more computing devices and to the point of sale device for display on the point of sale device, a message regarding approval of the transaction authorization request.

11. The method of claim 10, further comprising, receiving, from the computing device, an assignment of the identified user to the point of sale device at the location, wherein the payment account information are transmitted to the point of sale device based on the assignment.

12. A computer program product, comprising:

a non-transitory computer-readable medium having computer-readable program instructions embodied thereon that when executed by a computer cause the computer to identify users at locations by comparing facial imaging of users against facial templates of users known to be at particular locations, the computer-readable program instructions comprising;

computer-readable program instructions to broadcast an identifier associated with the computing device at a location;

computer-readable program instructions to receive, from one or more computing devices, one or more facial templates, each facial template associated with a user associated with a user computing device that receives the broadcasted identifier at the location and retransmits the identifier to the one or more computing devices, each facial template comprising a representation of a respective facial image of the respective user;

computer-readable program instructions to store the one or more received facial templates;

computer-readable program instructions to receive a request to identify a user;

computer-readable program instructions to capture, by a camera module, a video feed of an environment external to the computer;

computer-readable program instructions to extract a facial image of a particular user from the video feed;

computer-readable program instructions to generate a facial template of the particular user based on the facial image, wherein the facial template of the particular user is a representation of the facial image of the particular user;

computer-readable program instructions to retrieve the one or more stored facial templates associated with one or more corresponding user computing devices that retransmitted the identifier broadcasted at the location;

computer-readable program instructions to determine a similarity between the generated facial template of the particular user and each of the one or more stored facial templates;

computer-readable program instructions to identify a particular stored facial template as corresponding to the generated facial template of the particular user based on the similarity between the generated facial template of the particular user and the particular stored facial template exceeding a threshold; and

computer-readable program instructions to transmit a response to the request to identify the user comprising an indication of an identity of the particular user.

13. The computer program product of claim 12, wherein the environment external to the computer comprises the particular user located in proximity to one or more merchant system point of sale devices.

14. The computer program product of claim 12, wherein the request to identify a user is received from a merchant system point of sale device and wherein the response to the request to identify a user is transmitted to the merchant point of sale device.

15. The computer program product of claim 13, wherein the merchant point of sale device processes a transaction using payment account information associated with the identified particular user at a time after the response to the request to identify a user is transmitted to the merchant point of sale device.

16. The computer program product of claim 13, further comprising:

computer-readable program instructions to determine, based on a frame of the video feed, a position of a face of the particular user and positions of each of one or more point of sale devices at the merchant location;

computer-readable program instructions to assign, based on distances between the face of the particular user and each of the one or more point of sale devices in the frame of the video feed, the particular user to a particular point of sale device of the one or more point of sale devices,

wherein the indication of the identity of the particular user further comprises an identifier of the particular point of sale device and an indication of the assignment of the particular user to the particular point of sale device.

17. A system to identify users at locations by comparing facial imaging of users against facial templates of users known to be at particular locations, comprising:

a storage device; and

a processor communicatively coupled to the storage device, wherein the processor executes application code instructions that are stored in the storage device to cause the system to;

receive, from a particular user computing device, an account identifier of an account of a particular user associated with the particular user computing device and a beacon device identifier, wherein the particular user computing device retransmits the beacon device identifier received via a network connection from a beacon device at a location associated with the beacon device identifier;

retrieve a facial template associated with the account of the particular user based on the account identifier of the account of the particular user;

identify the location based on the beacon device identifier;

add the facial template associated with the account of the particular user and associated location to a current customer log comprising facial templates corresponding to user computing devices that retransmit the beacon device identifier received via the network connection from the beacon device at the location associated with the beacon device identifier; and

transmit, to a computing device at the location, the current customer log comprising the facial template associated with the account of the particular user, wherein the computing device identifies the particular user based on identifying the facial template associated with the account of the particular user from the current customer log as being similar to a facial template of the particular user generated from a facial image of the particular user captured by the computing device.

18. The system of claim 17, wherein the processor is further configured to execute computer-readable program instructions stored on the storage device to cause the system to:

determine that the particular user computing device no longer maintains a network connection with the beacon device;

in response to determining that the particular user computing device no longer maintains a network connection with the beacon device, remove the facial template associated with the account of the particular user and associated location from the current customer log to generate an updated current customer log; and

transmit, to the computing device, the updated current customer log.

19. The system of claim 17, wherein determining that the user computing device no longer maintains the network connection with the beacon device comprises detecting no subsequent transmission from the user computing device of the user account identifier and the beacon identifier within a threshold amount of time.

20. The system of claim 17, wherein the processor is further configured to execute computer-readable program instructions stored on the storage device to cause the system to:

receive, from the computing device, a facial template associated with an identified user, a transaction total, and a merchant account identifier from the computing device at the location;

determine, based on the received facial template associated with the identified user, an account identifier associated with the identified user;

transmit, to a point of sale device, payment account information associated with a user account associated with the account identifier associated with the identified user;

receive, from the point of sale device, an indication of a selection of particular payment account information from the payment account information for use in a transaction;

generate a transaction authorization request comprising the transaction total, the merchant account identifier, and the particular payment account information;

transmit the transaction authorization request to an issuer system associated with the particular payment account information;

receive, from the issuer system, data comprising an approval of the transaction authorization request; and

transmit, to the point of sale device for display on the point of sale device, a receipt comprising a notice of the approval of the transaction authorization request.