MACHINE, SYSTEM AND METHOD FOR USER-GUIDED TEACHING OF DEICTIC REFERENCES AND REFERENT OBJECTS OF DEICTIC REFERENCES TO A CONVERSATIONAL COMMAND AND CONTROL SYSTEM

• US 20100184011A1
• Filed: 01/21/2009
• Published: 07/22/2010
• Est. Priority Date: 01/21/2009
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Viswanathan, Mahesh, Comerford, Liam D.
• Granted Patent: US 9,311,917 B2
• Time in Patent Office: Days
• US Class Current: 434/321
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces

Abstract

A machine, system and method for user-guided teaching of deictic references and referent objects of deictic references to a conversational system. The machine includes a system bus for communicating data and control signals received from the conversational system to the computer system, a data and control bus for connecting devices and sensors in the machine, a bridge module for connecting the data and control bus to the system bus, respective machine subsystems coupled to the data and control bus, the respective machine subsystems having a respective user interface for receiving a deictic reference from a user, a memory coupled to the system bus for storing deictic references and objects of the deictic references learned by the conversational system and a central processing unit coupled to the system bus for executing the deictic references with respect to the objects of the deictic references learned.

Claims

1. A machine having a computer system for user-guided teaching of deictic references and objects of said deictic references to a conversational system in said machine, said machine comprising:

a conversational system installed on said computer system for carrying a dialog with a user, said conversational system including a deictic reference tool for decoding a deictic reference received from a user, wherein said deictic reference comprises of a combination of a physical pointing gesture and a spoken utterance, and wherein said machine executes said deictic reference with respect to one or more objects of said deictic reference decoded;

a system bus for communicating data and control signals received from said conversational system to said computer system;

a data and control bus for connecting devices and sensors in said machine;

a bridge module for connecting said data and control bus to said system bus;

one or more respective machine subsystems coupled to said data and control bus connected to said computer system in said machine, said one or more respective machine subsystems having a respective user interface for receiving said deictic reference from a user;

a memory coupled to said system bus for storing one or more deictic references and objects of said deictic references learned by said conversational system; and

a central processing unit coupled to said system bus for executing said deictic references with respect to said one or more objects of said deictic references learned.

2. The machine according to claim 1, wherein said deictic reference tool further comprises:

a verification module for confirming understanding of said one or more objects of said deictic reference learned from said user in order to detect and correct any errors pertaining to said one or more objects of said deictic reference learned, wherein said confirming understanding of said one or more objects of said deictic reference learned comprises one or more of;

displaying said object of said deictic reference utilizing a graphical user interface for permitting correction by spoken or graphical or deictic interactions, utilizing a laser scanner subsystem to outline or paint said object of said deictic reference, utilizing a computer-driven manipulator to point said object of said deictic reference.

3. The machine according to claim 2, wherein said deictic reference module observes said physical pointing gesture via one or more image capturing devices coupled to said computer system and decodes said one or more objects of said deictic reference utilizing said conversational system.

4. The machine according to claim 3, wherein said deictic reference tool further comprises:

a decoder module for decoding said physical pointing gesture to locate a reference region; and

an object recognition module for determining whether or not a object is a correct object that said physical pointing gesture points to.

5. The machine according to claim 4, wherein said conversational system further comprises:

a speech recognition module for decoding any spoken utterances received from a user using a respective spoken user interface of said one or more respective machine subsystems coupled to said data and control bus;

a gesture recognition module for decoding indications received via a respective physical user interface of said one or more respective machine subsystems coupled to said data and control bus; and

a haptic module for processing haptic interactions received via a respective physical user interface of said one or more respective machine subsystems coupled to said data and control bus.

6. A method for teaching one or more objects of a deictic reference to a machine having a conversational system, said method comprising the steps of:

invoking a dialog with said conversational system installed on said machine, said conversational system including a deictic reference tool for teaching one or more objects of a deictic reference to said machine, wherein said deictic reference comprises a combination of a physical pointing gesture and a voice command;

teaching said one or more objects of said deictic reference to said machine, wherein said machine performs one or more tasks with respect to said one or more objects of said deictic reference learned;

saving said one or more objects of said deictic reference confirmed; and

executing said deictic reference for performing said one or more tasks with respect to said objects of said deictic reference learned during execution of said deictic reference.

7. The method according to claim 6, wherein said teaching step further comprises the steps of:

decoding said physical pointing gesture of said deictic reference to obtain a reference region to locate an object of said deictic reference;

determining whether or not an image captured for a candidate object in said reference region is in fact said object of said deictic reference decoded by said conversational system;

adding said image captured for said object of said deictic reference to an image library; and

adding an object name for said object of said deictic reference in a vocabulary set of said conversational system.

8. The method according to claim 7, wherein said determining step further comprises the steps of:

recording responses received from said user for confirming whether or not said candidate object in said reference region is in fact said object of said deictic reference received from said user; and

if said candidate object is not said object, repeating decoding said physical pointing gesture of said deictic reference to obtain said reference region and to locate said object of said deictic reference.

9. The method according to claim 8, wherein said adding said image captured of said object step further comprises the steps of:

searching said image library for additional images of said object; and

training an image recognition module within said deictic reference tool to recognize said object utilizing said image captured and said additional images searched for said object.

10. The method according to claim 9, wherein said adding an object name for said object step further comprises the steps of:

building a command list for said deictic reference decoded based on said voice command decoded and said object of said deictic reference for execution of said deictic reference.

11. A computer program product for teaching one or more objects of a deictic reference to a machine having a conversational system, said computer program product comprising:

a computer readable storage medium;

first program instructions to initiate a dialog request with a deictic reference tool of said conversational system for teaching one or more objects of a deictic reference to said machine, wherein said deictic reference comprises a combination of a physical pointing gesture and a voice command;

second program instructions to teach said one or more objects of said deictic reference to said machine, said second program instructions including instructions to decode said deictic reference and to build a command list for executing said deictic reference decoded; and

third program instructions to execute said deictic reference for performing one or more tasks with respect to said one or more objects of said deictic reference learned, wherein said machine performs said one or more tasks with respect to said one or more objects of said deictic reference learned, and wherein said first, second and third program instructions are recorded on said computer readable storage medium.

12. The computer program product according to claim 11, further comprising:

fourth program instructions to record responses received from said user for confirming correct decoding of said deictic reference, wherein said confirming correct decoding of said deictic reference comprises one or more of;

displaying said object of said deictic reference utilizing a graphical user interface for permitting correction by spoken or graphical or deictic interactions, utilizing a laser scanner subsystem to outline or paint said object of said deictic reference, utilizing a computer-driven manipulator to point said object of said deictic reference, and wherein said fourth program instructions are recorded on said computer readable storage medium.

13. The computer program product according to claim 12, wherein said second program instructions further comprise instructions to decode said physical pointing gesture of said deictic reference to obtain a reference region to locate an object of said deictic reference, to determine whether or not an image captured for a candidate object in said reference region is in fact said object of said deictic reference, to add said image captured of said object of said deictic reference to an image library and to add an object name for said object in a vocabulary set of said conversational system.

14. The computer program product according to claim 13, wherein said second program instructions further comprise instructions to record responses received from said user for confirming whether or not said candidate object in said reference region is in fact said object of said deictic reference and, if said candidate object is not said object, to repeat decoding said physical pointing gesture of said deictic reference to obtain said reference region and to locate said object of said deictic reference.

15. The computer program product according to claim 14, wherein said second program instructions further comprise instructions to search said image library for additional images of said object and to train an image recognition module to recognize said object utilizing said image captured and said additional images searched for said object of said deictic reference.

16. A computer system for user-guided teaching of objects of said deictic reference to a conversational system, said computer system comprising:

first program instructions to initiate a dialog request with a deictic reference tool of said conversational system for teaching one or more objects of a deictic reference to said machine, wherein said deictic reference comprises a combination of a physical pointing gesture and a voice command;

second program instructions to teach said one or more objects of said deictic reference to said machine, said second program instructions including instructions to decode said deictic reference and to build a command list for executing said deictic reference decoded;

third program instructions to execute said deictic reference for performing one or more tasks with respect to said one or more objects of said deictic reference learned;

a computer readable storage medium for storing each of said first, second and third program instructions; and

a central processing unit for executing each of said first, second and third program instructions.

17. The computer system according to claim 16, further comprising:

fourth program instructions to record responses received from said user for confirming correct decoding of said deictic reference, wherein said confirming correct decoding of said deictic reference comprises one or more of;

displaying said object of said deictic reference utilizing a graphical user interface for permitting correction by spoken or graphical or deictic interactions, utilizing a laser scanner subsystem to outline or paint said object of said deictic reference, utilizing a computer-driven manipulator to point said object of said deictic reference, wherein said fourth program instructions are recorded on said computer readable storage medium.

18. The computer system according to claim 17, wherein said second program instructions further comprise instructions to decode said physical pointing gesture of said deictic reference to obtain a reference region to locate an object of said deictic reference, to determine whether or not an image captured for a candidate object in said reference region is in fact said object of said deictic reference, to add said image captured of said object of said deictic reference to an image library and to add an object name for said object in a vocabulary set of said conversational system.

19. The computer system according to claim 18, wherein said second program instructions further comprise instructions to record responses received from said user for confirming whether or not said candidate object in said reference region is in fact said object of said deictic reference and, if said object is not said object, to repeat decoding said physical pointing gesture of said deictic reference to obtain said reference region and to locate said object of said deictic reference.

20. The computer system according to claim 19, wherein said second program instructions further comprise instructions to search said image library for additional images of said object and to train an image recognition module to recognize said object utilizing said image captured and said additional images searched for said object of said deictic reference.

21. A process for deploying computing infrastructure comprising integrating computer-readable code into a computing system of a machine, wherein said code in combination with said computing system is capable of learning one or more objects of said deictic reference, said process comprising the steps of:

invoking a dialog with said conversational system installed on said machine, said conversational system including a deictic reference tool for teaching one or more objects of a deictic reference to said machine, wherein said deictic reference comprises a combination of a physical pointing gesture and a voice command;

teaching said one or more objects of said deictic reference to said machine, wherein said machine performs one or more tasks with respect to said one or more objects of said deictic reference learned;

saving said one or more objects of said deictic reference learned; and

executing said deictic reference for performing said one or more tasks with respect to said objects of said deictic reference learned during execution of said deictic reference.

22. The process according to claim 21, wherein said teaching step further comprises the steps of:

decoding said physical pointing gesture of said deictic reference to obtain a reference region to locate an object of said deictic reference;

determining whether or not an image captured for a candidate object in said reference region is in fact said object of said deictic reference decoded by said conversational system;

adding said image captured for said object of said deictic reference to an image library; and

adding an object name for said object of said deictic reference in a vocabulary set of said conversational system.

23. The process according to claim 22, wherein said determining step further comprises the steps of:

recording responses received from said user for confirming whether or not said candidate object in said reference region is in fact said object of said deictic reference received from said user; and

if said candidate object is not said object, repeating decoding said physical pointing gesture of said deictic reference to obtain said reference region and to locate said object of said deictic reference.

24. The process according to claim 23, wherein said adding said image captured of said object step further comprises the steps of:

searching said image library for additional images of said object; and

training an image recognition module within said deictic reference tool to recognize said object utilizing said image captured and said additional images searched for said object.

25. The process according to claim 24, wherein said adding an object name for said object step further comprises the steps of:

building a command list for said deictic reference decoded based on said voice command decoded and said object of said deictic reference for execution of said deictic reference.