SYSTEM AND METHOD FOR PROCESSING MULTI-MODAL DEVICE INTERACTIONS IN A NATURAL LANGUAGE VOICE SERVICES ENVIRONMENT

• US 20130304473A1
• Filed: 12/03/2012
• Published: 11/14/2013
• Est. Priority Date: 02/20/2009
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Baldwin, Larry, Weider, Chris
• Granted Patent: US 8,738,380 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A system and method for processing multi-modal device interactions in a natural language voice services environment may be provided. In particular, one or more multi-modal device interactions may be received in a natural language voice services environment that includes one or more electronic devices. The multi-modal device interactions may include a non-voice interaction with at least one of the electronic devices or an application associated therewith, and may further include a natural language utterance relating to the non-voice interaction. Context relating to the non-voice interaction and the natural language utterance may be extracted and combined to determine an intent of the multi-modal device interaction, and a request may then be routed to one or more of the electronic devices based on the determined intent of the multi-modal device interaction.

Claims

1-18. -18. (canceled)

19. A computer-implemented method of facilitating natural language utterance processing via multiple input modes, the method being implemented on a computer that includes one or more physical processors executing one or more computer program modules that perform the method, the method comprising:

receiving, via a first input mode, a first input;

receiving, via a second input mode that is different from the first input mode, a second input that relates to the first input;

determining a request based on the first input or the second input;

determining, based on the first input and the second input, context information for the request; and

processing the request based on the context information.

20. The method of claim 19, wherein determining the request comprises determining an action, a query, a command, or a task based on the first input or the second input.

21. The method of claim 19, wherein receiving the first input comprises receiving a natural language utterance via a voice input mode, and wherein receiving the second input comprises receiving a non-voice input via a non-voice input mode.

22. The method of claim 19, wherein receiving the first input comprises receiving a non-voice input via a non-voice input mode, and wherein receiving the second input comprises receiving a natural language utterance via a voice input mode.

23. The method of claim 22, wherein receiving the non-voice input comprises receiving the non-voice input via a first device having the non-voice input mode, and wherein receiving the natural language utterance comprises receiving the natural language utterance via a second device having the voice input mode.

24. The method of claim 22, wherein receiving the non-voice input comprises receiving the non-voice input via a first device having the non-voice input mode and the voice input mode, and wherein receiving the natural language utterance comprises receiving the natural language utterance via the first device.

25. The method of claim 22, wherein the non-voice input indicates an item, selected segment, operation, or point of focus on a first device, and wherein determining the context information comprises determining the context information based on the item, selected segment, operation, or point of focus on the first device.

26. The method of claim 22, wherein the non-voice input indicates a point of focus on a touch screen of a first device, and wherein determining the context information comprises determining a location associated with the point of focus on the touch screen.

27. The method of claim 26, wherein processing the request comprises determining a product or service based on the determined location, the method further comprising:

generating a response to the request based on the determined product or service.

28. The method of claim 26, wherein the point of focus includes a point or area on a display rendered on the touch screen, and wherein determining the location comprises determining a physical location based on the point or area on the display.

29. The method of claim 28, wherein the voice input indicates a product or service type, and wherein processing the request comprises determining a product or service based on the determined physical location and the product or service type, the method further comprising:

generating a response to the request based on the determined product or service.

30. The method of claim 22, wherein the natural language utterance indicates a domain associated with the request, and wherein processing the request comprises processing the request based on the domain and the context information.

31. The method of claim 22, further comprising:

determining a domain associated with the request based on the natural language utterance,

wherein processing the request comprises processing the request at a domain agent associated with the determined domain responsive to the determination of the domain.

32. The method of claim 22, wherein the request, the natural language utterance, or the non-voice input is associated with a user, the method further comprising:

determining one or more prior natural language utterances or prior non-voice inputs associated with the user,

wherein determining the context information for the request comprises determining the context information based on (i) the first input, (ii) the second input, and (iii) the one or more prior natural language utterances or prior non-voice inputs.

33. The method of claim 19, further comprising:

determining, based on a context stack having a plurality of entries that individually are indicative of context, an entry in the context stack that corresponds to the first input or the second input; and

determining a domain agent associated with the entry in the context stack, wherein processing the request comprises providing the request to the determined domain agent to process the request.

34. The method of claim 33, wherein the determined domain agent is configured to update the context stack responsive to the processing of the request by the determined domain agent.

35. A system for facilitating natural language utterance processing via multiple input modes, the system comprising one or more physical processors executing one or more computer program modules that cause the system to:

receive, via a first input mode, a first input;

receive, via a second input mode that is different from the first input mode, a second input that relates to the first input;

determine a request based on the first input or the second input;

determine, based on the first input and the second input, context information for the request; and

process the request based on the context information.

36. The system of claim 35, wherein determining the request comprises determining an action, a query, a command, or a task based on the first input or the second input.

37. The system of claim 35, wherein receiving the first input comprises receiving a natural language utterance via a voice input mode, and wherein receiving the second input comprises receiving a non-voice input via a non-voice input mode.

38. The system of claim 35, wherein receiving the first input comprises receiving a non-voice input via a non-voice input mode, and wherein receiving the second input comprises receiving a natural language utterance via a voice input mode.

39. The system of claim 38, wherein receiving the non-voice input comprises receiving the non-voice input via a first device having the non-voice input mode, and wherein receiving the natural language utterance comprises receiving the natural language utterance via a second device having the voice input mode.

40. The system of claim 38, wherein receiving the non-voice input comprises receiving the non-voice input via a first device having the non-voice input mode and the voice input mode, and wherein receiving the natural language utterance comprises receiving the natural language utterance via the first device.

41. The system of claim 38, wherein the non-voice input indicates an item, selected segment, operation, or point of focus on a first device, and wherein determining the context information comprises determining the context information based on the item, selected segment, operation, or point of focus on the first device.

42. The system of claim 38, wherein the non-voice input indicates a point of focus on a touch screen of a first device, and wherein determining the context information comprises determining a location associated with the point of focus on the touch screen.

43. The system of claim 42, wherein processing the request comprises determining a product or service based on the determined location, and wherein the system is caused to:

generate a response to the request based on the determined product or service.

44. The system of claim 42, wherein the point of focus includes a point or area on a display rendered on the touch screen, and wherein determining the location comprises determining a physical location based on the point or area on the display.

45. The system of claim 44, wherein the voice input indicates a product or service type, and wherein processing the request comprises determining a product or service based on the determined physical location and the product or service type, and wherein the system is caused to:

generate a response to the request based on the determined product or service.

46. The system of claim 38, wherein the natural language utterance indicates a domain associated with the request, and wherein processing the request comprises processing the request based on the domain and the context information.

47. The system of claim 38, wherein the system is caused to:

determine a domain associated with the request based on the natural language utterance, and

wherein processing the request comprises processing the request at a domain agent associated with the determined domain responsive to the determination of the domain.

48. The system of claim 38, wherein the request, the natural language utterance, or the non-voice input is associated with a user, and wherein the system is caused to:

determine one or more prior natural language utterances or prior non-voice inputs associated with the user, and

wherein determining the context information for the request comprises determining the context information based on (i) the first input, (ii) the second input, and (iii) the one or more prior natural language utterances or prior non-voice inputs.

49. The system of claim 35, wherein the system is caused to:

determine, based on a context stack having a plurality of entries that individually are indicative of context, an entry in the context stack that corresponds to the first input or the second input; and

determine a domain agent associated with the entry in the context stack, wherein processing the request comprises providing the request to the determined domain agent to process the request.

50. The system of claim 49, wherein the determined domain agent is configured to update the context stack responsive to the processing of the request by the determined domain agent.