Devices and methods for interfacing human users with electronic devices

• US 6,983,418 B1
• Filed: 05/30/2000
• Issued: 01/03/2006
• Est. Priority Date: 03/24/1995
• Status: Expired due to Fees

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Scott, Neil G.
• Primary Examiner(s): Kincaid, Kristine
• Assistant Examiner(s): Ke, Peng
• Application Number:
  US09/579,125
• Time in Patent Office: 2,044 Days
• Field of Search: 345/700, 345/726, 345/769, 345/768, 715/700, 715/726, 715/769, 715/768, 715771-773, 715761-763
• US Class Current: 715/700
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G07F 19/211 Software architecture within ATMs or in relation to the ATM network
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/08 Protocols for interworking; Protocol conversion
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method of interfacing human users with electronic devices liberates electronic devices from specific input and output devices and substitutes a universal communication system between them. The method combines the advantages of personal customization with the advantages of unlimited access to electronic devices. In particular, it provides disabled individuals with full access to suitably equipped electronic devices such as ATMs, telephones, fax machines, computers, copiers, TVs, VCRs, stereos, and microwave ovens. This method therefore provides the means for making these important electronic devices equally accessible to everyone. In a preferred embodiment, the communication system includes a speech synthesis and recognition system [56] which is connected to an accessor total access port [54], an ATM [70] connected to a target total access port [68], and a total access link [66] between the accessor total access port [54] and the target total access port [68]. The total access link [66] is an information link over which universal data packets are transmitted between the total access ports. The universal data packets have a device-independent form and contain user-functional representations of data. A dialog initiated by the accessor total access port [54] helps the user locate the ATM [70] and establish a total access link.

Claims

1. A method of controlling a target device using a communication system including an accessor total access port, an accessor device connected to said accessor total access port, a target total access port connected to said target device which is an unmodified electronic device, and a total access link between the accessor total access port and the target total access port, the method comprising:

converting, by said accessor device, user actions intended to control the target device into data of a device-dependent form specific to said accessor device, said data of the device-dependent form specific to the accessor device representing control of the target device, wherein the accessor device is a human user interface device that accepts intentions from the user in whatever form the user expresses them;

sending the data of the device-dependent form specific to the accessor device from the accessor device to the accessor total access port;

translating, by said accessor total access port, the data sent from the accessor device of the device-dependent form specific to the accessor device into data of a device-independent form containing a user-functional representation of the data sent from the accessor device;

transmitting the translated data of the device-independent form over the total access link from the accessor total access port to the target total access port;

translating, by said target total access port, the data sent from the accessor total access port of the device-independent form into data of a device-dependent form specific to the target device; and

sending the data of the device-dependent form specific to the target device from the target total access port to the target device,

wherein the target device is controlled by the data of the device-dependent form specific to the target device sent from the target total access port.

2. The method of claim 1, wherein the total access link comprises an information link chosen from the group including at least an infrared link, an electronic cable link, a computer network link, a fiber optic link, and a radio frequency link.

3. The method of claim 2, wherein the total access link comprises a bi-directional serial transmission link with error-handling, error-correction, handshaking, data packing, and data unpacking.

4. The method of claim 1, wherein the transmitting comprises creating a generic data packet chosen from the group including at least a keyboard packet, a mouse packet, a video packet, a target data packet, a control packet, and an accessor-to-accessor packet.

5. The method of claim 1, wherein the accessor device is selected from the group including at least a keyboard, a mouse, a track-ball, a joy-stick, a video monitor, a liquid-crystal display, an LED display, a speaker, a voice synthesizer, a speech recognition system, a remote control, a headband switch, a printer, sound generator, a Braille display, a tactile display, and a virtual reality display.

6. A method of controlling a target device using a communication system including an accessor total access port, an accessor device connected to said accessor total access port, a target total access port connected to said target device which is an unmodified electronic device, and a total access link between the accessor total access port and the target total access port, the method comprising:

converting user actions intended to control the target device into electronic data of a device-dependent form specific to the accessor device using the accessor device, wherein the accessor device is a human user interface device;

sending the electronic data of the device-dependent form specific to the accessor device from the accessor device to the accessor total access port;

translating the electronic data of the device-dependent form specific to the accessor device sent from the accessor device from the device-dependent form specific to the accessor device into a device-independent form containing a user-functional representation of the electronic data sent from the accessor device;

transmitting the translated electronic data of the device-independent form over the total access link from the accessor total access port to the target total access port;

translating the transmitted electronic data of the device-independent form into a device-dependent form specific to the target device; and

sending the translated electronic data of the device-dependent form specific to the target device from the target total access port to the target device,

wherein the target device is controlled by the data of the device-dependent form specific to the target device sent from the target total access port,

wherein the accessor device comprises;

a speech recognition and synthesis system, and

wherein the accessor total access port comprises;

a sound and voice interface for facilitating the use of speech input.

7. The method of claim 6, wherein the target device comprises a personal computer, and the target total access port comprises a ROM containing communication resource information about the personal computer.

8. A method of controlling a target device using a communication system comprising an accessor total access port, an accessor device connected to said accessor total access port, a target total access port connected to said target device which is an unmodified electronic device, and a total access link between the accessor total access port and the target total access port, the method comprising:

converting user actions intended to control the target device into electronic data of the device-dependent form specific to the accessor device using the accessor device, wherein the accessor device is a human user interface device;

sending the electronic data of the device-dependent form specific to the accessor device from the accessor device to the accessor total access port;

translating the electronic data of the device-dependent form specific to the accessor device sent from the accessor device from the device-dependent form specific to the accessor device into a device-independent form containing a user functional representation of the electronic data sent from the accessor device;

transmitting the translated electronic data of the device-independent form over the total access link from the accessor total access port to the target total access port;

translating the transmitted electronic data of the device-independent form into a device-dependent form specific to the target device;

sending the translated electronic data of the device-dependent form specific to the target device from the target total access port to the target device,

wherein the target device is controlled by the data of the device-dependent form specific to the target device sent from the target total access port; and

sending a radio frequency wake-up call from the accessor total access port to the target total access port.

9. An accessor total access port for permitting a user to control a target device in a system including an accessor device connected to the accessor total access port, the target device connected to a target total access port, said target device being an unmodified device, and a total access line between the accessor total access port and the target total access port, the accessor total access port comprising:

a microprocessor;

wherein said accessor device converts user actions intended to control the target device into data of a device-dependent form specific to said accessor device, said data of the device-dependent form specific to the accessor device representing control of the target device,

wherein the accessor device is a human user interface device that accepts intentions from the user in whatever form the user expresses them;

an accessor device interface between the microprocessor and the accessor device for communicating device-dependent electronic data specific to the accessor device between the accessor device and the microprocessor;

a memory containing data about the accessor device and instruction code including instructions for the microprocessor to translate the data of the device dependent form specific to the accessor device from the accessor device into data of a device-independent form containing a user-functional representation of the data from the accessor device; and

a communications interface between the microprocessor and the total access link for transmitting the data of the device-independent form over the total access link to the target total access port, thereby permitting the user to control the target device.

10. An accessor total access port for permitting a user to control a target device in a system including an accessor device connected to the accessor total access port, the target device connected to a target total access port, and a total access link between the accessor total access port and the target total access port, the accessor total access port comprising:

a microprocessor;

an accessor device interface between the microprocessor and the accessor device for communicating device-dependent electronic data between the accessor device and the microprocessor;

a memory containing data about the accessor device and instruction code including instructions for the microprocessor to translate the device-dependent electronic data from the accessor device into device-independent data containing a user-functional representation of the electronic data from the accessor device,

wherein the data about the accessor device includes device-dependent accessor data, device-independent link data, and user-functional descriptions; and

a communications interface between the microprocessor and the total access link for transmitting the device-independent data over the total access link to the target total access port, thereby permitting the user to control the target device,

wherein the communications interface comprises an infrared transceiver.

11. The accessor total access port of claim 10, wherein the infrared transceiver transmits universal data packet with error-handling, error-correction, handshaking, data packing, and data unpacking.

12. An accessor total access port for permitting a user to control a target device in a system including an accessor device connected to the accessor total access port, the target device connected to a target total access port, and a total access link between the accessor total access port and the target total access port, the accessor total access port comprising:

a microprocessor;

an accessor device interface between the microprocessor and the accessor device for communicating device-dependent electronic data between the accessor device and the microprocessor;

a memory containing data about the accessor device and instruction code including instructions for the microprocessor to translate the device-dependent electronic data from the accessor device into device-independent data containing a user-functional representation of the electronic data from the accessor device,

wherein the data about the accessor device includes device-dependent accessor data, device-independent link data, and user-functional descriptions; and

a communications interface between the microprocessor and the total access link for transmitting the device-independent data over the total access link to the target total access port, thereby permitting the user to control the target device,

wherein the microprocessor formats the device-independent data into a data packet chosen from the group including a keyboard packet, a mouse packet, a video packet, a target data packet, a control packet, and an accessor-to-accessor packet.

13. An accessor total access port for permitting a user to control a target device in a system including an accessor device connected to the accessor total access port, a target device connected to a target total access port, and a total access link between the accessor total access port and the target total access port, the accessor total access port comprising:

a microprocessor;

an accessor device interface between the microprocessor and the accessor device for communicating device-dependent electronic data between the accessor device and the microprocessor;

a memory containing data about the accessor device and instruction code including instructions for the microprocessor to translate the device-dependent electronic data from the accessor device into device-independent data containing a user-functional representation of the electronic data from the accessor device,

wherein the data about the accessor device includes device-dependent accessor data, device-independent link data, and user-functional descriptions; and

a communications interface between the microprocessor and the total access link for transmitting the device-independent data over the total access link to the target total access port, thereby permitting the user to control the target device,

wherein the accessor device comprises a speech recognition and synthesis system and the total access port comprises a sound and a voice interface for facilitating the use of speech input.

14. A system for permitting a human user to control an electronic device, the system comprising:

an accessor device interfaced directly with the user, wherein said accessor device converts user actions intended to control the electronic device into data of a device-dependent form specific to said accessor device, said data of the device dependent form specific to the accessor device representing control of the electronic device, wherein the accessor device is a human user interface device that accepts intentions from the user in whatever form the user expresses them;

a first microprocessor connected to the accessor device and programmed to translate between data of the device-dependent form specific to the accessor device and data of a device-independent form containing a user-functional representation that represents interactions of the user and the accessor device;

a second microprocessor connected to the electronic device and programmed to translate between the data of the device-independent form and data of a device dependent form specific to the electronic device,

wherein the electronic device is controlled by the data of a device-dependent form specific to the electronic device; and

a communication link connecting the first and second microprocessors,

wherein universal data packets are transmitted over the communication link between the first microprocessor and the second microprocessor each universal data packet including at least data of the device-independent form.

15. A method of permitting a human user to control an electronic device, the method comprising:

receiving input signals from an input/output device customized to the human user,

wherein said input/output device accepts intentions intended to control the electronic device from the user in whatever form the user expresses them and converts the user intentions into device-dependent data specific to the input/output device;

translating, by a first microprocessor, the input signals into device independent data containing a user-functional representation representing user intentions;

transmitting the device-independent data over a communications link from the first microprocessor to a second microprocessor;

translating, by the second microprocessor, the device-independent data into device-dependent data specific to the electronic device; and

sending the device-dependent data specific to the electronic device to the electronic device,

wherein the electronic device is controlled by the data of a device-dependent form specific to the electronic device.

16. The method of claim 15, further comprising transmitting a description of input/output device resources over the communications link.