Contextual auto-completion for assistant systems

• US 10,853,103 B2
• Filed: 10/02/2018
• Issued: 12/01/2020
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Zhu, Jiedan, Peng, Fuchun, Dumoulin, Benoit F., Liu, Xiaohu, Subba, Rajen, Agsen, Mohsen, Hanson, Michael Robert
• Primary Examiner(s): Morrison, Jay A
• Application Number:
  US16/150,069
• Publication Number:
  US 20190324780A1
• Time in Patent Office: 791 Days
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/3323 using document space presentation or visualization, e.g. category, hierarchy or range presentation and selection
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 18/2411 : based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 2216/13 : Prefetching
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 : Parsing
• CPC Class Codes: G06F 40/274 : Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/295 : Named entity recognition
• CPC Class Codes: G06F 40/30 : Semantic analysis
• CPC Class Codes: G06F 40/40 : Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 7/14 : Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
• CPC Class Codes: G06F 8/31 : Programming languages or programming paradigms
• CPC Class Codes: G06F 9/44505 : Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/4451 : User profiles; Roaming
• CPC Class Codes: G06F 9/453 : Help systems
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/006 : based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/027 : Frames
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 50/01 : Social networking
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/04 : Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/07 : to the speaker
• CPC Class Codes: G10L 15/16 : using artificial neural networks
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/183 : using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/187 : Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: H04L 12/2816 : Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 41/20 : Network management software packages
• CPC Class Codes: H04L 41/22 : comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/0882 : Utilisation of link capacity
• CPC Class Codes: H04L 43/0894 : Packet rate
• CPC Class Codes: H04L 5/02 : Channels characterised by the type of signal
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 : Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 : Commands or executable codes
• CPC Class Codes: H04L 51/216 : Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/222 : using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 : for supporting social networking services
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/53 : using third party service providers
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 67/75 : Indicating network or usage conditions on the user display
• CPC Class Codes: H04W 12/08 : Access security

Abstract

In one embodiment, a method includes receiving a user input including a partial request from a client system of a first user, analyzing the user input to generate one or more candidate hypotheses based on a personalized language model where each of the candidate hypotheses includes one or more of an intent-suggestion or a slot-suggestion, sending instructions for presenting one or more suggested auto-completions corresponding to one or more of the candidate hypotheses, respectively, to the client system, where each suggested auto-completion comprises the partial request and the corresponding candidate hypothesis, receiving an indication of a selection by the first user of a first suggested auto-completion of the suggested auto-completions from the client system, and executing one or more tasks based on the first suggested auto-completion selected by the first user via one or more agents.

Claims

1. A method comprising, by one or more computing systems:

receiving, from a client system associated with a first user, a first user input from the first user, wherein the first user input comprises a partial request;

sending, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent-suggestion, and wherein each suggested intent auto-completion comprises the partial request and the corresponding candidate hypothesis;

receiving, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input;

sending, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses corresponding to the second user input, respectively, wherein each of the one or more second candidate hypotheses comprise a slot-suggestion, and wherein each suggested slot auto-completion comprises the second user input and the corresponding candidate hypothesis;

receiving, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and

executing, via one or more agents, one or more tasks based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user.

2. The method of claim 1, further comprising analyzing, based on a personalized language model, the first user input to generate the one or more first candidate hypotheses corresponding to the partial request, wherein the analysis comprises:

analyzing, based on the personalized language model, the first user input to determine one or more candidate intents.

3. The method of claim 2, further comprising:

sending, to the client system, a request for additional information from the first user;

receiving, from the client system, an additional user input by the first user response to the request; and

disambiguating, based on the additional user input, the one or more candidate intents to determine a top candidate intent to provide as a suggested intent auto-completion for one of the first candidate hypotheses.

4. The method of claim 1, further comprising analyzing, based on a personalized language model, the second user input to generate the one or more second candidate hypotheses corresponding to the second user input, wherein the analysis comprises:

analyzing, based on the personalized language model, the second user input to determine one or more candidate slots.

5. The method of claim 4, further comprising:

sending, to the client system, a request for additional information by the first user;

receiving, from the client system, an additional user input by the first user response to the request; and

disambiguating, based on the additional user input, the one or more candidate slots to determine a top candidate slot to provide as a suggested slot auto-completion for one of the second candidate hypotheses.

6. The method of claim 2, wherein the personalized language model is trained based on a plurality of training data comprising one or more of:

newsfeed posts associated with the first user;

newsfeed comments associated with the first user;

messages in one or more messaging interfaces associated with the first user;

data characterizing one or more domains;

dialog states of one or more dialog sessions associated with the first user;

user profile data associated with the user;

or

task states associated with one or more tasks.

7. The method of claim 1, wherein the one or more first candidate hypotheses are ranked based on a dialog state of a dialog session associated with the first user input.

8. The method of claim 1, wherein the one or more first candidate hypotheses are associated with one or more confidence scores, respectively, wherein the one or more confidence scores are calculated by a personalized language model, and wherein the one or more first candidate hypotheses are ranked based on their respective confidence scores.

9. The method of claim 8, further comprising:

receiving, from the client system, an additional user input, wherein the additional user input is appended to the first user input;

updating, for the one or more first candidate hypotheses, the one or more confidence scores based on the additional user input; and

re-ranking the one or more first candidate hypotheses based on the updated confidence scores.

10. The method of claim 8, further comprising:

applying a sliding window to the first user input, wherein a length of the sliding window determines a percentage of the first user input to use as a model input to the personalized language model.

11. The method of claim 10, further comprising:

determining if at least one confidence score of the one or more confidence scores associated with the one or more first candidate hypothesis is smaller than a threshold score; and

upon determining that at least one confidence score is smaller than the threshold score, adjusting the length of the sliding window.

12. The method of claim 1, wherein analyzing the first user input to generate the one or more first candidate hypotheses corresponding to the partial request is further based on one or more context-specific language models.

13. The method of claim 12, further comprising:

accessing, by a dialog engine, a dialog state of a dialog session associated with the first user input;

selecting a particular context-specific language model from the one or more context-specific language models based on the dialog state; and

generating the one or more first candidate hypotheses based on a personalized language model and the selected context-specific model.

14. The method of claim 12, wherein the one or more context-specific language models are trained based on context-specific data comprising one or more of:

data associated with presences of the first user at particular locations;

data associated with interactions of the first user with particular users;

or

data associated with registrations of the first user at particular events.

15. The method of claim 1, further comprising analyzing, based on a personalized language model, the first user input to generate the one or more first candidate hypotheses corresponding to the partial request, wherein the analysis is further based on one or more global language models, wherein the one or more global language models are trained based on data associated with a plurality of users of an online social network.

16. The method of claim 1, further comprising analyzing, based on a personalized language model, the first user input to generate the one or more first candidate hypotheses corresponding to the partial request, wherein the analysis is further based on one or more global context-specific language models.

17. The method of claim 1, wherein each of the one or more first candidate hypotheses corresponds to a subsequent entry associated with the first user input.

18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive, from a client system associated with a first user, a first user input from the first user, wherein the first user input comprises a partial request;

send, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent-suggestion, and wherein each suggested intent auto-completion comprises the partial request and the corresponding candidate hypothesis;

receive, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input;

send, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses corresponding to the second user input, respectively, wherein each of the one or more second candidate hypotheses comprise a slot-suggestion, and wherein each suggested slot auto-completion comprises the second user input and the corresponding candidate hypothesis;

receive, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and

execute, via one or more agents, one or more tasks based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user.

19. A system comprising:

one or more processors; and

a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to;

receive, from a client system associated with a first user, a first user input from the first user, wherein the first user input comprises a partial request;

send, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent-suggestion, and wherein each suggested intent auto-completion comprises the partial request and the corresponding candidate hypothesis;

receive, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input;

send, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses corresponding to the second user input, respectively, wherein each of the one or more second candidate hypotheses comprise a slot-suggestion, and wherein each suggested slot auto-completion comprises the second user input and the corresponding candidate hypothesis;

receive, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and

execute, via one or more agents, one or more tasks based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user.