Multi-modal, geo-tempo communications systems

• US 8,275,834 B2
• Filed: 09/14/2009
• Issued: 09/25/2012
• Est. Priority Date: 09/14/2009
• Status: Active Grant

• Current Assignee: Applied Research Associates Incorporated
• Original Assignee: Applied Research Associates Incorporated
• Inventors: Aldunate, Roberto, Larson, Gregg E
• Primary Examiner(s): Moise, Emmanuel L
• Assistant Examiner(s): KOROBOV, VITALI A
• Application Number:
  US12/558,758
• Publication Number:
  US 20110066682A1
• Time in Patent Office: 1,107 Days
• Field of Search: 709/203, 709/204, 709/246, 706/47, 706/60, 701/400, 704/200, 707/607
• US Class Current: 709/204
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display

Abstract

Disclosed is a flexible, multi-modal system useful in communications among users, capable of synchronizing real world and augmented reality, wherein the system is deployed in centralized and distributed computational platforms. The system comprises a plurality of input devices designed and configured to generate signals representing speech, gestures, pointing direction, and location of a user, and transmit the same to a multi-modal interface. Some of the signals generated represent a message from the user intended for dissemination to other users. A plurality of agents and one or more databases (defined by matrices which include input commands or information from the input devices, and meanings for some input commands or combinations thereof) are integrated into the system, where at least some of the agents are designed and configured to receive signals from the multi-modal interface, translate the signals into data, compare the same to a database, generate signals representing meanings as defined by the database, and transmit the signals to the multi-modal interface. In some embodiments, at least one agent resides on a computational device located on a user, and at least one other agent resides on a computation device remote from the user. Finally, a plurality of output devices are associated with the system and designed and configured to receive and process signals from the multi-modal interface, some of said signals representing messages to the user to be communicated by means of an output device in visual, auditory, or tactile modes.

Claims

1. A flexible, multi-modal system useful in communications among users, capable of synchronizing real world and augmented reality, wherein the system is deployed in centralized and distributed computational platforms, the system comprising:

(a) a multi-modal interface comprising a sensor input interface and a sensor output interface, said multi-modal interface being designed and configured to receive and generate electronic or radio signals, said interface having a modifiable default mode for types of signals received and generated thereby;

(b) a plurality of input devices designed and configured to generate signals representing speech, gestures, pointing direction, and location of a user, and transmit the same to the multi-modal interface, wherein some of the signals generated represent a message from the user intended for dissemination to other users;

(c) a plurality of agents and one or more databases,

(i) wherein the databases are defined by matrices which include input commands or information from one or more input devices, and meanings for some input commands or combinations thereof;

(ii) wherein at least some of the agents are designed and configured to receive signals from the sensor input interface of the multi-modal interface, translate the signals into data, compare the same to a database to determine further action of the system, wherein said further action comprises;

(A) obtaining additional information from the user;

(B) sending a message to other users;

or (C) controlling an object designed and configured to receive signals from the system, and said at least some of the agents are further designed and configured to generate signals representing meanings as defined by the database, and transmit the signals to the multi-modal interface;

(iii) wherein at least one agent resides on a computational device located on a user, and at least one other agent resides on a computation device remote from the user; and

(iv) wherein at least one agent is a location-monitoring agent designed and configured to receive periodic signals representing location information of users, said signals being generated by an input device located on a user and transmitted to the multi-modal interface for further transmission to the location monitoring agent, and said agent is further designed and configured to monitor location changes of users, and in association with a database to take at least one of the following actions upon some location changes of users;

(1) generate messages for transmission to the multi-modal interface, for further transmission to one or more output devices; and

(2) modify the default mode of messages to another mode based upon the same or other location changes of users, such modification to be transmitted to the multi-modal interface to modify the default mode of messages generated thereby; and

(d) a plurality of output devices designed and configured to receive and process signals from the sensor output interface of the multi-modal interface, some of said signals representing messages to the user to be communicated by means of an output device in visual, auditory, or tactile modes.

2. The system of claim 1, wherein said multi-modal interface is designed and configured to generate electronic or radio signals by a plurality of transmission means, and wherein the plurality of agents further comprise a second agent designed and configured to monitor environmental conditions in a location of the system, to generate a signal representing an instruction to modify the transmission means of messages based upon said conditions, and transmitting said signal to the multi-modal interface.

3. The system of claim 1, wherein the plurality of agents further comprise a mapping agent, said agent comprising a geographical internal space being defined by relative local coordinates, based upon relation to a point set south-east of any point to be mapped;

said mapping agent being designed and configured to receive location information from a location sensor input of a user, and translate the same to system relative local coordinates.

4. The system of claim 3, wherein the plurality of agents further comprise a collision detection agent, said agent being coupled with a database comprising known objects or structures and their location in the geographical internal space, wherein said agent is designed and configured to receive orientation, azimuth and pitch information from the multi-modal interface or another agent, and location information from the multi-modal interface or the mapping agent, and using the location information and the direction information, determining which objects if any are intersected by a ray in the azimuth and pitch of the pointing device, originating from a vector of the user'"'"' s location at the point of orientation.

5. The system of claim 4, wherein the collision detection agent is further designed and configured to receive information from a range finder input device, and wherein said mapping agent stores an object in the database based upon range information from the range finder and the orientation, azimuth and pitch information from the multi-modal interface or another agent.

6. The system of claim 4 or 5, wherein object locations are organized into a search tree.

7. The system of claim 4 or 5, wherein objects may be tagged by a user by means of the collision detection agent and the mapping agent, the tagged objects being communicated to users by overlaying the outline of the object on a user'"'"'s visual screen, determined by the user'"'"'s look vector.

8. The system of claim 7, wherein tagged objects are first previewed and accepted or rejected by the tagging user before dissemination to other users.

9. The system of claim 1, wherein an agent is designed and configured to categorize messages and associate each message with the time it was generated;

the agent is further designed and configured to determine, based upon the message category of a message, the duration of time the message shall be communicated to other users.