Personalized gesture recognition for user interaction with assistant systems

• US 10,802,848 B2
• Filed: 04/18/2019
• Issued: 10/13/2020
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Liu, Xiaohu, Crook, Paul Anthony, Penov, Francislav P., Subba, Rajen
• Primary Examiner(s): Bolotin, Dmitriy
• Application Number:
  US16/388,130
• Publication Number:
  US 20190324553A1
• Time in Patent Office: 544 Days
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/3323 using document space presentation or visualization, e.g. category, hierarchy or range presentation and selection
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 18/2411 : based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 2216/13 : Prefetching
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 : Parsing
• CPC Class Codes: G06F 40/274 : Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/295 : Named entity recognition
• CPC Class Codes: G06F 40/30 : Semantic analysis
• CPC Class Codes: G06F 40/40 : Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 7/14 : Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
• CPC Class Codes: G06F 8/31 : Programming languages or programming paradigms
• CPC Class Codes: G06F 9/44505 : Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/4451 : User profiles; Roaming
• CPC Class Codes: G06F 9/453 : Help systems
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/006 : based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/027 : Frames
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 50/01 : Social networking
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/04 : Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/07 : to the speaker
• CPC Class Codes: G10L 15/16 : using artificial neural networks
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/183 : using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/187 : Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: H04L 12/2816 : Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 41/20 : Network management software packages
• CPC Class Codes: H04L 41/22 : comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/0882 : Utilisation of link capacity
• CPC Class Codes: H04L 43/0894 : Packet rate
• CPC Class Codes: H04L 5/02 : Channels characterised by the type of signal
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 : Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 : Commands or executable codes
• CPC Class Codes: H04L 51/216 : Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/222 : using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 : for supporting social networking services
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/53 : using third party service providers
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 67/75 : Indicating network or usage conditions on the user display
• CPC Class Codes: H04W 12/08 : Access security

Abstract

In one embodiment, a method includes accessing a plurality of input tuples associated with a first user from a data store, wherein each input tuple comprises a gesture-input and a corresponding speech-input, determining a plurality of intents corresponding to the plurality of speech-inputs, respectively, by a natural-language understanding (NLU) module, generating a plurality of feature representations for the plurality of gesture-inputs based on one or more machine-learning models, determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations, associating the plurality of intents with the plurality of gesture identifiers, respectively, and training a personalized gesture-classification model for the first user based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

Claims

1. A method comprising, by one or more computing systems:

accessing, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;

determining, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;

generating, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;

determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;

associating the plurality of intents with the plurality of gesture identifiers, respectively; and

training, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

2. The method of claim 1, further comprising:

accessing, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

3. The method of claim 2, wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

4. The method of claim 1, further comprising:

generating, by one or more automatic speech recognition (ASR) modules, a plurality of text-inputs for the plurality of speech-inputs, respectively.

5. The method of claim 4, wherein determining the plurality of intents corresponding to the plurality of speech-inputs, respectively, is based on the plurality of text-inputs of the respective speech-inputs.

6. The method of claim 1, wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

7. The method of claim 1, wherein the personalized gesture-classification model is based on convolutional neural networks.

8. The method of claim 1, wherein generating each feature representation for each gesture-input comprises:

dividing the gesture-input into one or more components; and

modeling the one or more components into the feature representation for the gesture-input.

9. The method of claim 1, wherein generating each feature representation for each gesture-input comprises:

determining temporal information associated with the gesture-input; and

modeling the temporal information into the feature representation for the gesture-input.

10. The method of claim 1, further comprising:

receiving, from a client system associated with the first user, a new gesture-input from the first user; and

determining, for the new gesture-input, an intent corresponding to the new gesture-input based on the personalized gesture-classification model.

11. The method of claim 10, further comprising:

executing one or more tasks based on the determined intent.

12. The method of claim 1, wherein training the personalized gesture-classification model is further based on user feedback data from the first user.

13. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;

determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;

generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;

determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;

associate the plurality of intents with the plurality of gesture identifiers, respectively; and

train, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

14. The media of claim 13, wherein the software is further operable when executed to:

access, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

15. The media of claim 14, wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

16. The media of claim 13, wherein the software is further operable when executed to:

generate, by one or more automatic speech recognition (ASR) modules, a plurality of text-inputs for the plurality of speech-inputs, respectively.

17. The media of claim 16, wherein determining the plurality of intents corresponding to the plurality of speech-inputs, respectively, is based on the plurality of text-inputs of the respective speech-inputs.

18. The media of claim 13, wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

19. The media of claim 13, wherein the personalized gesture-classification model is based on convolutional neural networks.

20. A system comprising:

one or more processors; and

a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to;

access, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;

determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;

generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;

determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;

associate the plurality of intents with the plurality of gesture identifiers, respectively; and

train, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.