Methods and systems for multi-modal browsing and implementation of a conversational markup language

• US 7,685,252 B1
• Filed: 04/06/2000
• Issued: 03/23/2010
• Est. Priority Date: 10/12/1999
• Status: Expired due to Term

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Raman, Thiruvilvama Lai V., Maes, Stephane Herman
• Primary Examiner(s): Blair; Douglas B
• Application Number:
  US09/544,823
• Time in Patent Office: 3,638 Days
• Field of Search: 709/203, 709217-220, 709/310, 709/315, 709/246, 709/248, 704/270, 704/275, 379/88, 379/17, 717/173, 715/513, 715/236
• US Class Current: 709/217
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 2203/355 Interactive dialogue design tools, features or methods
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

A new application programming language is provided which is based on user interaction with any device which a user is employing to access any type of information. The new language is referred to herein as a “Conversational Markup Language (CML). In a preferred embodiment, CML is a high level XML based language for representing “dialogs” or “conversations” the user will have with any given computing device. For example, interaction may comprise, but is not limited to, visual based (text and graphical) user interaction and speech based user interaction. Such a language allows application authors to program applications using interaction-based elements referred to herein as “conversational gestures.” The present invention also provides for various embodiments of a multimodal browser capable of supporting the features of CML in accordance with various modality specific representations, e.g., HTML based graphical user interface (GUI) browser, VoiceXML based speech browser, etc.

Claims

1. A method of generating an application accessible by a user through one or more computer-based devices, the method comprising the steps of:

representing interactions that the user is permitted to have with the one or more computer-based devices used to access the application by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application, and further wherein the interaction-based programming components may be transcoded on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modality-specific browsers associated with the one or more computer-based devices, such that the interaction-based programming components are independent of any modality and any modality-specific browser; and

authoring the application using at least a portion of the interaction-based programming components;

wherein representation by the interaction-based programming components permits synchronization of the one or more modality-specific renderings of the application on the one or more computer-based devices.

2. The method of claim 1, in a client/server arrangement wherein at least a portion of the application is to be downloaded from a server to at least one of the one or more computer-based devices, acting as a client, further comprising the step of including code in the application operative to provide a connection to the content/application logic resident at the server.

3. The method of claim 2, wherein the code in the application operative to provide a connection to the content/application logic expresses at least one of one or more data models, attribute constraints and validation rules associated with the application.

4. The method of claim 1, wherein the one or more modality-specific renderings comprise a speech-based representation of portions of the application.

5. The method of claim 4, wherein the speech-based representation is based on VoiceXML.

6. The method of claim 1, wherein the one or more modality-specific renderings comprise a visual-based representation of portions of the application.

7. The method of claim 6, wherein the visual-based representation is based on at least one of HTML, CHTML and WML.

8. The method of claim 1, wherein the user interactions are declaratively represented by the interaction-based programming components.

9. The method of claim 1, wherein the user interactions are imperatively represented by the interaction-based programming components.

10. The method of claim 1, wherein the user interactions are declaratively and imperatively represented by the interaction-based programming components.

11. The method of claim 1, wherein the interaction-based programming components comprise basic elements associated with a dialog that may occur between the user and the one or more computer-based devices.

12. The method of claim 11, wherein the interaction-based programming components comprise complex elements, the complex elements being aggregations of two or more of the basic elements associated with the dialog that may occur between the user and the one or more computer-based devices.

13. The method of claim 1, wherein one of the interaction-based programming components represent conversational gestures.

14. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating informational messages to the user.

15. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating contextual help information.

16. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating actions to be taken upon successful completion of another gesture.

17. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating yes or no based questions.

18. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating dialogues where the user is expected to select from a set of choices.

19. The method of claim 18, wherein the select gesture comprises a subelement that represents the set of choices.

20. The method of claim 18, wherein the select gesture comprises a subelement that represents a test that the selection should pass.

21. The method of claim 20, wherein the select gesture comprises a subelement that represents an error message to be presented if the test fails.

22. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating rules for validating results of a given conversational gesture.

23. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating grammar processing rules.

24. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating dialogues that help the user navigate through portions of the application.

25. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating a request for at least one of user login and authentication information.

26. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating a request for constrained user input.

27. The method of claim 13, wherein the conversational gestures comprise a gesture for encapsulating a request for unconstrained user input.

28. The method of claim 13, wherein the conversational gestures comprise a gesture for controlling submission of information.

29. The method of claim 1, further comprising the step of defining logical input events and an association between the logical input events and physical input events that trigger the defined logical input events, such that the application may be authored using at least a portion of the definitions.

30. The method of claim 1, wherein the component by component transcoding is performed in accordance with XSL transformation rules.

31. The method of claim 1, wherein the component by component transcoding is performed in accordance with Java Bean.

32. The method of claim 1, wherein the component by component transcoding is performed in accordance with Java Server Pages.

33. The method of claim 1, wherein representation by the interaction-based programming components supports a natural language understanding environment.

34. The method of claim 1, further comprising the step of including code in the authored application for permitting cosmetic altering of a presentational feature associated with the one or more modality-specific renderings of the application on the one or more computer-based devices.

35. The method of claim 1, further comprising the step of including code in the authored application for permitting changes to rules for transcoding on a component by component basis to generate the one or more modality-specific renderings of the application on the one or more computer-based devices.

36. The method of claim 1, wherein a definition of an underlying data model being populated is separated from a markup language defining the user interaction.

37. The method of claim 1, wherein an author is provided with a pass through mechanism to encapsulate modality-specific markup components.

38. The method of claim 1, wherein the components may be active in parallel.

39. The method of claim 1, wherein the representation and transcoding is extensible.

40. The method of claim 1, wherein a state of the application is encapsulated.

41. The method of claim 1, wherein the representation permits reference to dynamically generated data and supports callback mechanisms to the content/application logic.

42. A method of generating an application accessible by a user through one or more computer-based devices the method comprising the steps of:

representing interactions that the user is permitted to have with the one or more computer-based devices used to access the application by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application, and further wherein the interaction-based programming components may be transcoded on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modality-specific browsers associated with the one or more computer-based devices, such that the interaction-based programming components are independent of any modality and any modality-specific browser; and

authoring the application using at least a portion of the interaction-based programming components;

wherein a node_id attribute is attached to each component and the attribute is mapped over to various outputs.

43. Apparatus for use in accessing an application in association with one or more computer-based devices, the apparatus comprising:

one or more processors operative to;

(i) obtain the application from an application server, the application being programmatically represented by interactions that the user is permitted to have with the one or more computer-based devices by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application; and

(ii) transcode the interaction-based programming components on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modaility-specific browsers associated with the one or more computer-based devices, the interaction-based programming components being independent of any modality and any modality-specific browser

wherein representation by the interaction-based programming components permits synchronization of the one or more modality-specific renderings of the application on the one or more computer-based devices.

44. The apparatus of claim 43, wherein the one or more processors are distributed over the one or more computer-based devices.

45. The apparatus of claim 43, in a client/server arrangement wherein at least a portion of the application is to be downloaded from a server to at least one of the one or more computer-based devices, acting as a client, further comprising the step of including code in the application operative to provide a connection to the content/application logic resident at the server.

46. The apparatus of claim 45, wherein the code in the application operative to provide a connection to the content/application logic expresses at least one of one or more data models, attribute constraints and validation rules associated with the application.

47. The apparatus of claim 43, wherein the one or more modality-specific renderings comprise a speech-based representation of portions of the application.

48. The apparatus of claim 47, wherein the speech-based representation is based on VoiceXML.

49. The apparatus of claim 43, wherein the one or more modality-specific renderings comprise a visual-based representation of portions of the application.

50. The apparatus of claim 49, wherein the visual-based representation is based on at least one of HTML, CHTML and WML.

51. The apparatus of claim 43, wherein the user interactions are declaratively represented by the interaction-based programming components.

52. The apparatus of claim 43, wherein the user interactions are imperatively represented by the interaction-based programming components.

53. The apparatus of claim 43, wherein the user interactions are declaratively and imperatively represented by the interaction-based programming components.

54. The apparatus of claim 43, wherein the interaction-based programming components comprise basic elements associated with a dialog that may occur between the user and the one or more computer-based devices.

55. The apparatus of claim 54, wherein the interaction-based programming components comprise complex elements, the complex elements being aggregations of two or more of the basic elements associated with the dialog that may occur between the user and the one or more computer-based devices.

56. The apparatus of claim 43, wherein one of the interaction-based programming components represent conversational gestures.

57. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating informational messages to the user.

58. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating contextual help information.

59. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating actions to be taken upon successful completion of another gesture.

60. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating yes or no based questions.

61. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating dialogues where the user is expected to select from a set of choices.

62. The apparatus of claim 61, wherein the select gesture comprises a subelement that represents the set of choices.

63. The apparatus of claim 61, wherein the select gesture comprises a subelement that represents a test that the selection should pass.

64. The apparatus of claim 63, wherein the select gesture comprises a subelement that represents an error message to be presented if the test fails.

65. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating rules for validating results of a given conversational gesture.

66. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating grammar processing rules.

67. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating dialogues that help the user navigate through portions of the application.

68. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating a request for at least one of user login and authentication information.

69. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating a request for constrained user input.

70. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for encapsulating a request for unconstrained user input.

71. The apparatus of claim 56, wherein the conversational gestures comprise a gesture for controlling submission of information.

72. The apparatus of claim 43, further comprising the step of defining logical input events and an association between the logical input events and physical input events that trigger the defined logical input events, such that the application may be authored using at least a portion of the definitions.

73. The apparatus of claim 43, wherein the component by component transcoding is performed in accordance with XSL transformation rules.

74. The apparatus of claim 43, wherein the component by component transcoding is performed in accordance with Java Bean.

75. The apparatus of claim 43, wherein the component by component transcoding is performed in accordance with Java Server Pages.

76. The apparatus of claim 43, wherein representation by the interaction-based programming components supports a natural language understanding environment.

77. The apparatus of claim 43, further comprising the step of including code in the authored application for permitting cosmetic altering of a presentational feature associated with the one or more modality-specific renderings of the application on the one or more computer-based devices.

78. The apparatus of claim 43, further comprising the step of including code in the authored application for permitting changes to rules for transcoding on a component by component basis to generate the one or more modality-specific renderings of the application on the one or more computer-based devices.

79. The apparatus of claim 43, wherein a definition of an underlying data model being populated is separated from a markup language defining the user interaction.

80. The apparatus of claim 43, wherein an author is provided with a pass through mechanism to encapsulate modality-specific markup components.

81. The apparatus of claim 43, wherein the components may be active in parallel.

82. The apparatus of claim 43, wherein the representation and transcoding is extensible.

83. The apparatus of claim 43, wherein a state of the application is encapsulated.

84. The apparatus of claim 43, wherein the representation permits reference to dynamically generated data and supports callback mechanisms to the content/application logic.

85. The apparatus of claim 43, wherein the representation of the application further permits cosmetization of the one or more modality-specific renderings via one or more modality-specific markup languages.

86. Apparatus for use in accessing an application in association with one or more computer-based devices, the apparatus comprising:

one or more processors operative to;

(i) obtain the application from an application server, the application being programmatically represented by interactions that the user is permitted to have with the one or more computer-based devices by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application; and

(ii) transcode the interaction-based programming components on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modaility-specific browsers associated with the one or more computer-based devices, the interaction-based programming components being independent of any modality and any modality-specific browser;

wherein a node_id attribute is attached to each component and the attribute is mapped over to various outputs.

87. Apparatus for use in accessing an application in association with one or more computer-based devices, the apparatus comprising:

one or more processors operative to;

(i) obtain the application from an application server, the application being programmatically represented by interactions that the user is permitted to have with the one or more computer-based devices by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application; and

(ii) transcode the interaction-based programming components on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modaility-specific browsers associated with the one or more computer-based devices, the interaction-based programming components being independent of any modality and any modality-specific browser;

wherein the one or more processors are distributed over the one or more computer-based devices and the application is synchronized across the one or more computer-based devices.

88. A browser apparatus for use in providing access to an application by a user through one or more computer-based devices, comprising a machine readable storage medium containing computer executable code which when executed permits the implementation of the steps of:

obtaining the application from an application server, the application being programmatically represented by interactions that the user is permitted to have with the one or more computer-based devices by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application; and

transcoding the interaction-based programming components on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modaility-specific browsers associated with the one or more computer-based devices, the interaction-based programming components being independent of any modality and any modality-specific browsers

wherein representation by the interaction-based programming components permits synchronization of the one or more modality-specific renderings of the application on the one or more computer-based devices.

89. An article of manufacture for use in generating an application accessible by a user through one or more computer-based devices, comprising a machine readable storage medium containing computer executable code which when executed permits the implementation of the steps of:

representing interactions that the user is permitted to have with the one or more computer-based devices used to access the application by interaction-based programming components, wherein the interaction-based programming components are independent of content/application logic and presentation requirements associated with the application, and further wherein the interaction-based programming components may be transcoded on a component by component basis to generate one or more modality-specific renderings of the application renderable in accordance with one or more modality-specific browsers associated with the one or more computer-based devices, such that the interaction-based programming components are independent of any modality and any modality-specific browser; and

authoring the application using at least a portion of the interaction-based programming components;

wherein representation by the interaction-based programming components permits synchronization of the one or more modality-specific renderings of the application on the one or more computer-based devices.