Systems, methods, and computer program products for location salience modeling for multimodal search
First Claim
1. A method, for performing a multimodal search, comprising:
- receiving, by a multimodal search system having a processor, from a remote mobile device, a query information package comprising;
(I) a search query text string comprising a search topic component and a location component, the location component comprising a type of information selected from a first group consisting of;
(a) precise location information;
(b) ambiguous location information; and
(c) no location information;
(II) map state information comprising boundary information of a map displayed on the remote mobile device when the search query text string was issued and zoom-level information indicating a zoom level of the map when the search query text string was issued;
(III) touch tracing input information when the search query text string was issued;
(IV) prior query location information, if available;
(V) history of recent map movements information; and
(VI) history of recent utterances information;
determining which type of information, of the first group, the location component includes;
if it is determined that the location component comprises one or both of (b) ambiguous location information and (c) no location information;
determining, by the multimodal search system, based at least partially upon a decision tree model trained prior to the multimodal search system receiving the query information package, a search location in which the multimodal search system should search for the search topic component, including determining the search location to be one of a second group consisting of;
a first specific location determined using (II) the map state information when the search query text string was issued;
a second specific location determined using (III) the touch tracing input information when the search query text string was issued;
a third specific location determined using (IV) the prior query location information;
a fourth specific location determined using (V) the history of recent map movements history information; and
a fifth specific location determined using (VI) the history of recent utterances information;
determining, by the multimodal search system, a set of search results based upon the search location and the search topic component; and
sending the set of search results to the remote mobile device;
training the decision tree model, comprising;
receiving, by the multimodal search system, a plurality of training query information packages from a training remote mobile device, each training query information package comprising;
a training search query text string comprising a training search topic component and a training location component, the training location component comprising one of training ambiguous location information and no training location information;
training map state information;
training touch tracing input information;
training prior query location information;
training map manipulation information; and
training geographic location information;
instructing the training remote mobile device from which at least one of the plurality of training query information packages is received to provide a disambiguation interface configured to request an intended location of the training location component via a plurality of selectable options, the selectable options comprising a current location, a currently displayed location, a last spoken location, and a last touched location;
receiving, by the multimodal search system, the intended location from the training remote mobile device from which at least one of the plurality of training query information packages is received as one of the current location, the currently displayed location, the last spoken location, and the last touched location;
storing the intended locations in combination with the training map state information, the training touch tracing input information, the training prior query location information, the training map manipulation information, and the training geographic location information to create a decision tree instance for each of the intended locations;
training, using the decision tree instances, the decision tree model; and
performing, until a threshold value of decision tree instances is at least reached, operations to create the decision tree instances.
4 Assignments
0 Petitions
Accused Products
Abstract
Computational models of dialog context have often focused on unimodal spoken dialog or text, using the language itself as the primary locus of contextual information. But as spoken unimodal interaction is replaced by situated multimodal interaction on mobile platforms supporting a combination of spoken dialog with graphical interaction, touch-screen input, geolocation, and other non-linguistic contextual factors, a need arises for more sophisticated models of context that capture the influence of these factors on semantic interpretation and dialog flow. The systems, methods, and computer program products disclosed herein address this need. A method for multimodal search includes, in part, determining an intended location of search query based upon information received from a remote mobile device that issued the search query.
-
Citations
11 Claims
-
1. A method, for performing a multimodal search, comprising:
-
receiving, by a multimodal search system having a processor, from a remote mobile device, a query information package comprising; (I) a search query text string comprising a search topic component and a location component, the location component comprising a type of information selected from a first group consisting of; (a) precise location information; (b) ambiguous location information; and (c) no location information; (II) map state information comprising boundary information of a map displayed on the remote mobile device when the search query text string was issued and zoom-level information indicating a zoom level of the map when the search query text string was issued; (III) touch tracing input information when the search query text string was issued; (IV) prior query location information, if available; (V) history of recent map movements information; and (VI) history of recent utterances information; determining which type of information, of the first group, the location component includes; if it is determined that the location component comprises one or both of (b) ambiguous location information and (c) no location information; determining, by the multimodal search system, based at least partially upon a decision tree model trained prior to the multimodal search system receiving the query information package, a search location in which the multimodal search system should search for the search topic component, including determining the search location to be one of a second group consisting of; a first specific location determined using (II) the map state information when the search query text string was issued; a second specific location determined using (III) the touch tracing input information when the search query text string was issued; a third specific location determined using (IV) the prior query location information; a fourth specific location determined using (V) the history of recent map movements history information; and a fifth specific location determined using (VI) the history of recent utterances information; determining, by the multimodal search system, a set of search results based upon the search location and the search topic component; and sending the set of search results to the remote mobile device; training the decision tree model, comprising; receiving, by the multimodal search system, a plurality of training query information packages from a training remote mobile device, each training query information package comprising; a training search query text string comprising a training search topic component and a training location component, the training location component comprising one of training ambiguous location information and no training location information; training map state information; training touch tracing input information; training prior query location information; training map manipulation information; and training geographic location information; instructing the training remote mobile device from which at least one of the plurality of training query information packages is received to provide a disambiguation interface configured to request an intended location of the training location component via a plurality of selectable options, the selectable options comprising a current location, a currently displayed location, a last spoken location, and a last touched location; receiving, by the multimodal search system, the intended location from the training remote mobile device from which at least one of the plurality of training query information packages is received as one of the current location, the currently displayed location, the last spoken location, and the last touched location; storing the intended locations in combination with the training map state information, the training touch tracing input information, the training prior query location information, the training map manipulation information, and the training geographic location information to create a decision tree instance for each of the intended locations; training, using the decision tree instances, the decision tree model; and performing, until a threshold value of decision tree instances is at least reached, operations to create the decision tree instances. - View Dependent Claims (2, 3)
-
-
4. A non-transitory computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to perform operations comprising:
-
receiving, from a remote mobile device, a query information package comprising; (I) a search query text string comprising a search topic component and a location component, the location component comprising a type of information selected from a first group consisting of; (a) precise location information; (b) ambiguous location information; and (c) no location information; (II) map state information comprising boundary information of a map displayed on the remote mobile device when the search query text string was issued and zoom-level information indicating a zoom level of the map when the search query text string was issued; (III) touch tracing input information when the search query text string was issued; (IV) prior query location information, if available; (V) history of recent map movements information; and (VI) history of recent utterances information; determining which type of information, of the first group, the location component includes; if the processor determines that the location component comprises one or both of (b) ambiguous location information and (c) no location information; determining, based at least partially upon a decision tree model trained prior to receiving the query information package, a search location in which to search for the search topic component, including determining the search location to be one of a second group consisting of; a first specific location determined using (II) the map state information when the search query text string was issued; a second specific location determined using (III) the touch tracing input information when the search query text string was issued; a third specific location determined using (IV) the prior query location information; a fourth specific location determined using (V) the history of recent map movements history information; and a fifth specific location determined using (VI) the history of recent utterances information; determining a set of search results based upon the search location and the search topic component; and sending the set of search results to the remote mobile device; training the decision tree model, comprising; receiving a plurality of training query information packages from a training remote mobile device, each training query information package comprising; a training search query text string comprising a training search topic component and a training location component, the training location component comprising one of training ambiguous location information and no training location information; training map state information; training touch tracing input information; training prior query location information; training map manipulation information; and training geographic location information; instructing the training remote mobile device from which at least one of the plurality of training query information packages is received to provide a disambiguation interface configured to request an intended location of the training location component via a plurality of selectable options, the selectable options comprising a current location, a currently displayed location, a last spoken location, and a last touched location; receiving the intended location from the training remote mobile device from which at least one of the plurality of training query information packages is received as one of the current location, the currently displayed location, the last spoken location, and the last touched location; storing the intended locations in combination with the training map state information, the training touch tracing input information, the training prior query location information, the training map manipulation information, and the training geographic location information to create a decision tree instance for each of the intended locations; training, using the decision tree instances, the decision tree model; and performing, until a threshold value of decision tree instances is at least reached, operations to create the decision tree instances. - View Dependent Claims (5, 6)
-
-
7. A multimodal search system comprising:
-
a processor; and a memory, communicative with the processor, comprising computer-executable instructions that, when executed by the processor, cause the processor to perform operations comprising; receiving, from a remote mobile device, a query information package comprising; (I) a search query text string comprising a search topic component and a location component, the location component comprising a type of information selected from a first group consisting of; (a) precise location information; (b) ambiguous location information; and (c) no location information; (II) map state information comprising boundary information of a map displayed on the remote mobile device when the search query text string was issued and zoom-level information indicating a zoom level of the map when the search query text string was issued; (III) touch tracing input information when the search query text string was issued; (IV) prior query location information, if available; (V) history of recent map movements information; and (VI) history of recent utterances information; determining which type of information, of the first group, the location component includes; performing, if the processor determines that the location component comprises one or both of (b) ambiguous location information and (c) no location information, additional operations comprising; determining, based at least partially upon a decision tree model trained prior to receiving the query information package, a search location in which the multimodal search system should search for the search topic component, including determining the search location to be one of a second group consisting of; a first specific location determined using (II) the map state information when the search query text string was issued; a second specific location determined using (III) the touch tracing input information when the search query text string was issued; a third specific location determined using (IV) the prior query location information; a fourth specific location determined using (V) the history of recent map movements history information; and a fifth specific location determined using (VI) the history of recent utterances information; determining a set of search results based upon the search location and the search topic component; and sending, via a communications interface, the set of search results to the remote mobile device;
training the decision tree model, comprising;receiving a plurality of training query information packages from a training remote mobile device, each training query information package comprising; a training search query text string comprising a training search topic component and a training location component, the training location component comprising one of training ambiguous location information and no training location information; training map state information; training touch tracing input information; training prior query location information; training map manipulation information; and training geographic location information; instructing the training remote mobile device from which at least one of the plurality training query information packages is received to provide a disambiguation interface configured to request an intended location of the training location component via a plurality of selectable options, the selectable options comprising a current location, a currently displayed location, a last spoken location, and a last touched location; receiving the intended location from the training remote mobile device from which at least one of the plurality of training query information packages is received as one of the current location, the currently displayed location, the last spoken location, and the last touched location; storing the intended locations in combination with the training map state information, the training touch tracing input information, the training prior query location information, the training map manipulation information, and the training geographic location information to create a decision tree instance for each of the intended locations; training, using the decision tree instances, the decision tree model; and performing, until a threshold value of decision tree instances is at least reached, operations to create the decision tree instances. - View Dependent Claims (8, 9, 10, 11)
-
Specification