SYSTEM AND METHOD FOR PROCESSING MULTI-MODAL DEVICE INTERACTIONS IN A NATURAL LANGUAGE VOICE SERVICES ENVIRONMENT

• US 20130054228A1
• Filed: 09/14/2012
• Published: 02/28/2013
• Est. Priority Date: 02/20/2009
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Weider, Chris, BALDWIN, LARRY
• Granted Patent: US 8,719,009 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A system and method for processing multi-modal device interactions in a natural language voice services environment may be provided. In particular, one or more multi-modal device interactions may be received in a natural language voice services environment that includes one or more electronic devices. The multi-modal device interactions may include a non-voice interaction with at least one of the electronic devices or an application associated therewith, and may further include a natural language utterance relating to the non-voice interaction. Context relating to the non-voice interaction and the natural language utterance may be extracted and combined to determine an intent of the multi-modal device interaction, and a request may then be routed to one or more of the electronic devices based on the determined intent of the multi-modal device interaction.

Claims

1-18. -18. (canceled)

19. A method for processing a natural language utterance via multiple input modes, comprising:

receiving, at one or more processors, a first input via a first input mode and a second input via a second input mode, wherein the first input mode is different than the second input mode, wherein one of the first input or the second input identifies a query and the other of the first or second input provides context information for the query;

identifying the query from the one of the first input or the second input;

identifying the context information from the other of the first input or the second input; and

generating a response to the identified query based on the context information for the query.

20. The method of claim 19, wherein the receiving the first input comprises receiving a natural language utterance via a microphone and receiving the second input comprises receiving a non-voice input via another input device.

21. The method of claim 20, wherein the natural language utterance identifies the query, wherein the second input provides context information for one or more words of the natural language utterance, and wherein the generating the response is based on the context information for the one or more words of the natural language utterance.

22. The method of claim 21, wherein the one or more words includes a reference word, and wherein the second input provides context information for the reference word.

23. The method of claim 21, wherein the natural language utterance identifies a domain of the query, and wherein the generating the response is based on the domain and based on the context information.

24. The method of claim 20, wherein the non-voice input identifies data, a segment, item, or a point of focus on the another input device, wherein the data, segment, item, or point of focus provides context information for the natural language utterance.

25. The method of claim 20, wherein the second input identifies a point of focus on a touch screen, wherein the identifying the context information comprises identifying a location based on the point of focus on the touch screen.

26. The method of claim 25, wherein the query requests a product or service, and wherein the generating the response comprises retrieving information on the product or service at the location identified by the second input.

27. The method of claim 20, wherein the non-voice input identifies the query, and wherein the natural language utterance provides context information for the query.

28. The method of claim 27, wherein the query has a predetermined parameter, wherein the identifying the context information comprises identifying a value of the predetermined parameter from one or more words of the natural language utterance.

29. The method of claim 27, wherein the receiving the first input and the second input comprises:

detecting the non-voice input; and

receiving, in response to the detecting the non-voice input, the natural language utterance.

30. The method of claim 19, wherein the generating the response to the identified query comprises routing the query to a device or application configured to process process the query.

31. The method of claim 20, wherein the identifying the query from the one of the first input or the second input comprises identifying a wh-word in the natural language utterance, the wh-word including an interrogative pronoun.

32. The method of claim 19, further comprising:

receiving, at the one or more processors, a third input;

identifying second context information from the third input, the context information different than the second context information;

generating a response to the third input based on the identified query and based on the second context information.

33. The method of claim 32, wherein the one of the first input or the second input identifies a domain of the query, wherein the other of the first input or the second input identifies a location associated with the query, and wherein the third input identifies a different location associated with the query.

34. A device for processing a natural language utterance via multiple input modes, comprising:

one or more processors configured to;

receive a first input via a first input mode and a second input via a second input mode, wherein the first input mode is different than the second input mode, wherein one of the first input or the second input identifies a query and the other of the first or second input provides context information for the query,

identify the query from the one of the first input or the second input;

identify the context information from the other of the first input or the second input; and

generate a response to the identified query based on the context information for the query.

35. A non-transitory computer-readable medium for processing a natural language utterance via multiple input modes, the non-transitory computer-readable medium comprising one or more instructions that, when executed by one or more processors, cause the one or more processors to:

receive a first input via a first input mode and a second input via a second input mode, wherein the first input mode is different than the second input mode, wherein one of the first input or the second input identifies a query and the other of the first or second input provides context information for the query,

identify the query from the one of the first input or the second input;

identify the context information from the other of the first input or the second input; and

generate a response to the identified query based on the context information for the query.

36. A method for processing a natural language utterance via multiple input modes, comprising:

receiving, at one or more processors, a first input via a first input mode and a second input via second input mode, wherein the first input mode is different than the second input mode, wherein one of the first input or the second input identifies a command, and the other of the first or second input provides context information for the command;

identifying the command from the one of the first input or the second input;

identifying the context information from the other of the first input or the second input; and

generating a command signal based on the context information for the query.

37. The method of claim 36, wherein the receiving the first input comprises receiving a natural language utterance via a microphone and receiving the second input comprises receiving a non-voice input via another input device, and wherein the natural language utterance identifies a multimedia processing command.

38. The method of claim 37, wherein the context information identifies multimedia associated with the multimedia processing command.