Input commands via visual cues

• US 10,488,940 B2
• Filed: 03/09/2018
• Issued: 11/26/2019
• Est. Priority Date: 03/09/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Jouhikainen, Hannes, Sherif, Timur, Blanchet, Steve
• Primary Examiner(s): Lee, Ting Z
• Application Number:
  US15/916,809
• Publication Number:
  US 20190278377A1
• Time in Patent Office: 627 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 40/125 Finance or payroll
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream

Abstract

Embodiments disclosed herein generally relate to a method and system of generating text input via facial recognition. A computing system receives a video stream of a user operating an application on a client device. The video stream includes a time series of images of the user. The computing system parses the video stream to identify one or more visual cues of the user. The computing system identifies a current page of the application. The computing system maps the identified on or more visual cues to an operation available on the current page of the application. The computing system executes the mapped operation.

Claims

1. A method, comprising:

receiving, from a client device, at a management computing system remote from the client device, a request for a web page of a website hosted by a web server of the management computing system, the request received via an application executing on the client device;

transmitting, to the client device, the web page associated with the request;

requesting, by the management computing system from the client device, permission for the application executing on the client device to access a camera associated with the client device;

receiving, from the client device, permission for the application to access the camera while a user of the client device operates the application executing on the client device;

receiving, from a camera associated with the client device, at the management computing system, a video stream of a user interacting with the web page of the website via the application executing on the client device, the video stream comprising a time series of images of the user;

parsing, by the management computing system, the video stream to identify one or more visual cues of the user;

identifying a current page of the website accessed by the user via the application by parsing a uniform resource locator associated with the request for the web page;

mapping the identified one or more visual cues to an operation available on the current page of the website; and

executing the mapped operation.

2. The method of claim 1, wherein parsing the video stream to identify one or more visual cues of the user, comprises:

identifying lip movement of the user over the time series of images to identify one or more words mouthed by the user.

3. The method of claim 1, wherein parsing the video stream to identify one or more visual cues of the user, comprises:

determining one or more gestures of the user performed by the user over the time series of images.

4. The method of claim 1, wherein mapping the identified one or more visual cues to an operation available on the current page of the website, comprises:

referencing a database storing predefined visual cues linked to operations of the website.

5. The method of claim 4, wherein the predefined visual cues are user specific.

6. The method of claim 1, wherein the one or more visual cues of the user maps to the operation while accessing the current page of the website, and wherein the one or more visual cues of the user maps to a second operation while accessing a second page of the website.

7. The method of claim 1, wherein mapping the identified one or more visual cues to an operation available on the current page of the website, comprises:

mapping the identified, one or more visual cues to an authentication operation that grants the user access to a profile of the user on the website.

8. A system, comprising:

a processor; and

memory storing program code, which, when executed by the processor, performs an operation, comprising;

receiving, from a remote client device, a request for a web page of a website hosted by a web server of the system, the request received via an application executing on the client device;

transmitting, to the client device, the web page associated with the request;

requesting, from the client device, permission for the application executing on the client device to access a camera associated with the client device;

receiving, from the client device, permission for the application to access the camera while a user of the client device operates the application executing on the client device;

receiving, from the client device, a video stream of a user accessing the web page hosted by the web server via the application executing on the client device;

parsing a time series of images in the video stream to identify a command visually communicated by the user;

identifying a current page of the website accessed by the user via the application by parsing a uniform resource identifier associated with the request for the web page;

mapping the identified visual command of the user to an operation available on the current page of the website; and

executing the mapped operation.

9. The system of claim 8, wherein parsing the time series of images in the video stream to identify a command visually communicated by the user, comprises:

identifying one or more visual cues performed by the user.

10. The system of claim 9, wherein the one or more visual cues of the user maps to the operation while accessing the current page of the website, and wherein the one or more visual cues of the user maps to a second operation while accessing a second page of the website.

11. The system of claim 8, wherein parsing the time series of images in the video stream to identify a command visually communicated by the user, comprises:

determining a lip movement of the user to identify one or more words mouthed by the user.

12. The system of claim 8, wherein parsing the time series of images in the video stream to identify a command visually communicated by the user, comprises:

determining one or more gestures performed by the user.

13. The system of claim 8, wherein mapping the identified visual command to an operation available on the current page of the website, comprising:

referencing a database storing predefined visual cues linked to operations of the application.

14. The system of claim 13, wherein the predefined visual cues are user specific.

15. The system of claim 8, wherein mapping the identified visual command to an operation available on the current page of the website, comprises:

mapping the identified visual command to an authentication operation that grants the user access to a profile of the user on the application.

16. The system of claim 8, wherein the operation is developed from a training data set that was developed using a training algorithm.

17. A method, comprising:

requesting, by a client device, a web page of a website hosted by a remote server, the request sent via an application executing on the client device;

receiving, by the client device from the remote server, the web page associated with the request;

receiving, by the client device from the remote server, a request for the application executing on the client device to access a camera associated with the client device;

granting, by the client device to the application, access to the camera associated with the client device while a user of the client device is operating the application executing on the client device;

capturing, by the camera associated with the client device, a video stream of a user while the user is operating the application on the client device;

parsing, by the client device, the video stream to identify one or more visual cues to identify a command visually communicated by the user;

mapping, by the client device, the one or more visual cues performed by the user to one or more stored commands;

sending, by the client device, the mapped command to the remote server to identify an operation the stored command maps to on a current page of the website by identifying a uniform resource locator associated with the request for the web page;

receiving, by the client device, the operation from the remote server; and

updating, by the client device, an interface of the client device to prompt the user with a text-based description of the operation.

18. The method of claim 17, further comprising:

determining that the user has rejected the text-based description of the operation; and

prompting the user to capture a further video stream of the user while operating the application.

19. The method of claim 17, further comprising:

determining that the user has affirmed the text-based description of the operation; and

notifying the remote server that the user has affirmed the operation.

20. The method of claim 17, wherein parsing the video stream to identify one or more visual cues to identify a command visually communicated by the user, comprises:

identifying one or more lip movements of the user.