Interactive manual, system and method for vehicles and other complex equipment

• US 7,720,680 B2
• Filed: 06/17/2004
• Issued: 05/18/2010
• Est. Priority Date: 06/17/2004
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Schmidt, Hauke, Weng, Fuliang, Bei, Gengyan
• Primary Examiner(s): Hudspeth; David R
• Assistant Examiner(s): Jackson; Jakieda R
• Application Number:
  US10/871,249
• Publication Number:
  US 20050283363A1
• Time in Patent Office: 2,161 Days
• Field of Search: 704/235, 704/275
• US Class Current: 704/235
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method and system of providing an interactive manual, including a speech engine to receive and process speech from a user, convert the speech into a word sequence, and identify meaning structures from the word sequence, a structured manual including information related to an operation of a device, a visual model to relate visual representation of the information, a dialog management arrangement to interpret the meaning structures in a context and to extract pertinent information and the visual representation from the structured manual and the visual model, and an output arrangement to output the information and visual representation.

Claims

1. A method of interacting with a user to provide information regarding an operation of a device, comprising:

receiving speech input from the user;

converting the speech into a word sequence;

identifying a structure to which the word sequence conforms;

providing a structured manual including information related to an operation of a device;

providing a visual model to relate a visual representation of the information;

interpreting the structure in a context and extracting information and associated visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning category into which the structure of the word sequence has been categorized; and

outputting the information to the user,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

what-is”

meaning structure category causes the dialog arrangement to;

determine an object in the “

what-is”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the what-is table based on the object identification for a textual description associated with a feature of the object; and

display the textual description when the associated feature is displayed.

2. The method of claim 1, wherein the structure is identified using an automatic speech recognition arrangement and a natural language understanding arrangement.

3. The method of claim 1, wherein the information of the structured manual is arranged based on a human cognition structure.

4. The method of claim 1, wherein the visual graphic representation is one of vehicle-specific and device-specific.

5. The method of claim 1, further comprising:

receiving further input from the user.

6. The method of claim 5, wherein the further input includes a question regarding the information output to the user.

7. The method of claim 5, wherein the further input includes a cursor movement.

8. The method of claim 5, wherein the further input includes a cursor movement and a question regarding the information output to the user.

9. The method of claim 5, further comprising:

interrupting the output of information to the user upon receiving the further input from the user;

converting the further input from the user into further structures;

extracting information and associated visual representation related to the further structures from the structured manual and the visual model;

outputting further information to the user; and

resuming the output of information to the user at a point of interruption.

10. A storage medium having a set of instructions executable by a processor to perform a method of interacting with a user to provide information regarding an operation of a device, the method comprising:

receiving speech input from the user;

converting the speech into a word sequence;

identifying a structure to which the word sequence conforms;

providing a structured manual including information related to an operation of a device;

providing a visual model to relate a visual representation of the information;

interpreting the structure in a context and extracting information and associated visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning category into which the structure of the word sequence has been categorized; and

outputting the information to the user,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

what-is”

meaning structure category causes the dialog arrangement to;

determine an object in the “

what-is”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the what-is table based on the object identification for a textual description associated with a feature of the object; and

display the textual description when the associated feature is displayed.

11. An interactive manual system, comprising:

a speech engine to receive and process speech from a user, convert the speech into a word sequence, and identify from the word sequence at least one of a spatial meaning structure, a geometric meaning structure, and a color meaning structure from the word sequence;

a structured manual including textual information related to an operation of a device;

a visual model to relate portions of a visual representation of the device to portions of the textual information;

a dialog management arrangement that uses a context to identify at least one component of the device based on the identified at least one meaning structure and that extracts at least one portion of the textual information and at least one portion of the visual representation that pertain to the identified at least one component, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category; and

an output arrangement to output the extracted portions of the textual information and the visual representation,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

what-is”

meaning structure category causes the dialog arrangement to;

determine an object in the “

what-is”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the what-is table based on the object identification for a textual description associated with a feature of the object; and

display the textual description when the associated feature is displayed.

12. An interactive manual system, comprising:

a speech engine to receive and process speech from a user, convert the speech into a word sequence, and identify a structure to which the word sequence conforms;

a structured manual including information related to an operation of a device;

a visual model to relate a visual representation of the information;

a dialog management arrangement to interpret the structure in a context and to extract pertinent information and the visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning structure category into which the structure of the word sequence has been categorized; and

an output arrangement to output the information and visual representation,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

what-is”

meaning structure category causes the dialog arrangement to;

determine an object in the “

what-is”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the what-is table based on the object identification for a textual description associated with a feature of the object; and

display the textual description when the associated feature is displayed.

13. The interactive manual system of claim 12, wherein the speech engine includes an automatic speech recognition arrangement and a natural language understanding arrangement.

14. The interactive manual system of claim 13, wherein the natural language understanding arrangement identifies the structure from one of statistical and rule-based knowledge.

15. The interactive manual system of claim 12, wherein the device includes at least one of a vehicle and a device that resides in the vehicle.

16. The interactive manual system of claim 12, wherein the information of the structured manual is arranged based on a human cognition structure.

17. The interactive manual system of claim 12, wherein the visual representation is one of vehicle-specific and device-specific.

18. The interactive manual system of claim 12, wherein the output arrangement includes a visual display and a speech synthesis arrangement.

19. The interactive manual system of claim 18, wherein the visual display includes at least one of a 2-D and a 3-D display.

20. The interactive manual system of claim 18, wherein the visual display includes multiple levels.

21. The interactive manual system of claim 18, wherein the visual display includes an animation of an operation.

22. The interactive manual system of claim 12, wherein the speech synthesis arrangement includes a text-to-speech (TTS) application.

23. The interactive manual system of claim 12, wherein the output arrangement provide at least one of verbal and visual cues to guide the user.

24. The interactive manual system of claim 12, wherein the dialog management arrangement interprets the structure based on an interaction description language.

25. The interactive manual system of claim 12, wherein the object file includes a surrounding table including an object data of at least one object to define at least one of spatial and graphical information of at least one adjacent object to the at least one object, and the object data of the at least one object includes directional information of the at least one adjacent object relative to a position of the at least one object.

26. The interactive manual system of claim 25, wherein the directional information includes information indicating at least one of a left direction, a right direction, an upwards direction, a downwards direction, a forward direction, and a backward direction.

27. The interactive manual system of claim 25, wherein the object data of the at least one object further includes information according to which a likelihood of pertinence of a surrounding object to the speech from the user is determined.

28. The interactive manual system of claim 12, wherein the system is configured for modular construction of the model package to include a plurality of device models with a common presentation format for the plurality of device models.

29. An interactive manual system, comprising:

a speech engine to receive and process speech from a user, convert the speech into a word sequence, and identify a structure to which the word sequence conforms;

a structured manual including information related to an operation of a device;

a visual model to relate a visual representation of the information;

a dialog management arrangement to interpret the structure in a context and to extract pertinent information and the visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning structure category into which the structure of the word sequence has been categorized; and

an output arrangement to output the information and visual representation,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

how-to”

meaning structure category causes the dialog arrangement to;

determine an object in the “

how-to”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the slide table for animation clips associated with the object identification;

link the animation clips associated with the object identification and the animation clips in the grammar table; and

display the linked animation clips.

30. The interactive manual system of claim 29, wherein the system is configured for modular construction of the model package to include a plurality of device models with a common presentation format for the plurality of device models.

31. An interactive manual system, comprising:

a speech engine configured for converting an utterance from a user into a word sequence and identifying a meaning structure category to which the word sequence conforms based on one of a statistical method and rule-based method, a determination being made whether the utterance is a command or a question, the meaning structure category being identified as either a “

how-to”

meaning structure category or a “

what-is”

meaning structure category;

a display manager arrangement configured for displaying views of a current object based on the command;

a model manager arrangement configured for storing manual data regarding at least one object, and an object file, the manual data including text descriptions, static display frames, animation clips, and a lookup table for searching the manual data, the object file including at least one of spatial and graphical information of the object; and

an interaction manager arrangement, programmed with a plurality of rule and action pairs, configured for controlling a display via the display manager arrangement and based on an identified meaning structure category;

wherein the interaction manager arrangement identifies an object in one of the identified meaning structure categories, determines a rule based on the identified object and the identified meaning structure category, and generates a display via the display manager arrangement based on manual data and object file data associated with the identified object and the identified meaning structure category.

32. The interactive manual system of claim 31, wherein the manual data include a grammar package and a grammar table defined using a grammar specification language.

33. The interactive manual system of claim 32, wherein the objects file includes a model of at least one device component.

34. The interactive manual system of claim 33, wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table.

35. The interactive manual system of claim 34, wherein the grammar package includes a set of phrases and sentences as grammars for the speech engine to recognize, and the grammar table includes animation clips associated with the specified grammars.

36. A method of interacting with a user to provide information regarding an operation of a device, comprising:

receiving speech input from the user;

converting the speech into a word sequence;

identifying a structure to which the word sequence conforms;

providing a structured manual including information related to an operation of a device;

providing a visual model to relate a visual representation of the information;

interpreting the structure in a context;

extracting information and associated visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning category into which the structure of the word sequence has been categorized; and

outputting the information to the user,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

how-to”

meaning structure category causes the dialog arrangement to;

determine an object in the “

how-to”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the slide table for animation clips associated with the object identification;

link the animation clips associated with the object identification and the animation clips in the grammar table; and

display the linked animation clips.

37. A storage medium having a set of instructions executable by a processor to perform a method of interacting with a user to provide information regarding an operation of a device, the method comprising:

receiving speech input from the user;

converting the speech into a word sequence;

identifying a structure to which the word sequence conforms;

providing a structured manual including information related to an operation of a device;

providing a visual model to relate a visual representation of the information;

interpreting the structure in a context;

extracting information and associated visual representation from the structured manual and the visual model, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category, each of the meaning structure categories characterizing the structures of its members as being of a single respective underlying meaning structure, different information being deemed pertinent for the extraction depending upon the meaning category into which the structure of the word sequence has been categorized; and

outputting the information to the user,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

how-to”

meaning structure category causes the dialog arrangement to;

determine an object in the “

how-to”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the slide table for animation clips associated with the object identification;

link the animation clips associated with the object identification and the animation clips in the grammar table; and

display the linked animation clips.

38. An interactive manual system, comprising:

a speech engine to receive and process speech from a user, convert the speech into a word sequence, and identify from the word sequence at least one of a spatial meaning structure, a geometric meaning structure, and a color meaning structure from the word sequence;

a structured manual including textual information related to an operation of a device;

a visual model to relate portions of a visual representation of the device to portions of the textual information;

a dialog management arrangement that uses a context to identify at least one component of the device based on the identified at least one meaning structure and that extracts at least one portion of the textual information and at least one portion of the visual representation that pertain to the identified at least one component, wherein the structure is categorized into one of a “

how-to”

meaning structure category and a “

what-is”

meaning structure category; and

an output arrangement to output the extracted portions of the textual information and the visual representation,

wherein the structure manual and the visual model form a model package that includes a grammar package and a grammar table defined using a grammar specification language,

wherein the model package further includes an objects file to define device model information that includes a model of at least one device component,

wherein the object file is associated with at least one of a surrounding table, a slide table, a how-to table, and a what-is table,

wherein the grammar package includes a set of phrases and sentences as grammars for the speech recognition engine to recognize, and the grammar table includes animation clips associated with the specified grammars, and

wherein a categorization into the “

how-to”

meaning structure category causes the dialog arrangement to;

determine an object in the “

how-to”

meaning structure;

determine an object identification of the object;

search the surrounding table for information of adjacent objects based on the object identification;

search the slide table for animation clips associated with the object identification;

link the animation clips associated with the object identification and the animation clips in the grammar table; and

display the linked animation clips.