Spatial asset management system that time-tags and combines captured speech data and captured location data using a predifed reference grammar with a semantic relationship structure

• US 6,272,457 B1
• Filed: 09/16/1996
• Issued: 08/07/2001
• Est. Priority Date: 09/16/1996
• Status: Expired due to Term

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Datria Systems Incorporated (Honeywell International Inc.)
• Inventors: Yotka, John Anthony, Ford, Terry Edward, Turek, Richard James Jr.
• Primary Examiner(s): Thomas, Joseph
• Application Number:
  US08/714,583
• Time in Patent Office: 1,786 Days
• Field of Search: 704/9, 704/10, 704/1, 704/270, 704/276, 704/275, 707/100, 707/101, 707/104, 455/456, 701/207, 701/208, 701/213, 701/214, 701/215, 342/357.01, 342/357.06, 342/357.07, 342/357.08, 342/357.09, 342/357.1, 342/357.12, 342/357.13
• US Class Current: 704/9
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A data collection and automatic database population system which combines global positioning system (GPS), speech recognition software, radio frequency (RF) communications and geographic information system (GIS) to allow rapid capture of field data, asset tracking and automatic transfer of the data to a GIS database. A pre-defined grammar allows observations to be continuously captured, GPS location and time to be captured and stored on the field unit. Other sensor data is combined with the observations and combined with GPS and time information. A mobile unit'"'"'s location is tracked real time or post processed through wireless RF transmission of location information between the mobile unit and a central processing station. Real time position correction is provided by Differential GPS (DGPS). The captured data is electronically transferred to a central processing station for operator performed quality assurance and automatic population of the GIS database. The system provides the algorithms to automatically correlate the data to linear and point observations in the GIS database. The system algorithms provide for automatic correlation of field data with all other GIS database layers. Tools to generate predefined or user defined reports, Work orders and general data queries allow exploitation of the GIS database.

Claims

1. A method of combining captured location data with captured voice data by using a predefined reference grammar with a semantic relationship structure, said method comprising:

constructing the predefined reference grammar for the voice data to be captured;

incorporating the semantic relationship structure in the predefined reference grammar;

capturing free speech stating verbal observations containing the voice data;

time tagging each said captured free speech stating verbal observations to create a raw verbal observation;

capturing the location data contemporaneously with each said captured free speech stating verbal observations;

time tagging the captured location data; and

using the predefined reference grammar and the semantic relationship structure to enable the captured voice data in at least two raw verbal observations to be combined automatically with the captured location data into a single record in a geographic information system data base.

2. The invention defined in claim 1 including time tagging all records in the geographic information system data base using only the central processing unit clock of a computer.

3. The invention defined in claim 2 including capturing a third stream of sensor data;

time tagging all said captured third stream of sensor data by said central processing unit clock of said computer; and

automatically combining specific items of said third stream of sensor data into specific records having specific voice data and specific location data by using said time tagging as a synchronizing key.

4. The invention defined in claim 1 including using a mobile computer configured with a global positioning satellite receiver to capture the location data.

5. The invention defined in claim 1 including using a mobile computer configured with a sound board for capturing said free speech stating verbal observations containing the voice data.

6. The invention defined in claim 1 including using a mobile computer for capturing both the location data and said free speech stating verbal observations containing the voice data and including transferring the captured location data and the captured voice data from said mobile computer to a processing computer which combines the captured voice data and the captured location data into said single record which is stored in said geographic information system data base associated with said processing computer.

7. The invention defined in claim 1 including configuring the predefined reference grammar and the semantic relationship structure to a specific form for a specific application.

8. The invention defined in claim 7 wherein said specific application is a street maintenance application and the predefined reference grammar and the semantic relationship structure are configured to a specific form for use in said street maintenance application.

9. The invention defined in claim 1 including correlating the captured location data to linear networks.

10. The invention defined in claim 1 including correlating the captured location data to point observations.

11. The invention defined in claim 1 including correlating the location data to each said raw verbal observation and then to certain layers of said geographic information system data base.

12. A spatial asset management system apparatus for combining captured location data with captured voice data by using a reference grammar with a semantic relationship structure, said apparatus comprising:

a reference grammar means for interpreting the captured voice data contained in a verbal observation;

said reference grammar means incorporating a semantic relationship structure means for combining the captured voice data contained in at least two of said verbal observations;

free speech capturing means for capturing said verbal observation containing the voice data;

location data capturing means for capturing the location data contemporaneously with said verbal observation;

time tagging means for time tagging said captured verbal observation to create a raw verbal observation and for also time tagging the captured location data; and

data conversion processing means for using said reference grammar means and said semantic relationship structure means to enable the voice data in at least two of said raw verbal observations to be combined automatically with the captured location data into a single record in a geographic information system data base.

13. The invention defined in claim 12 wherein said time tagging means uses only the central processing unit clock of a computer.

14. The invention defined in claim 13 including a sensor data capturing means for capturing a third stream of sensor data, and wherein said time tagging means time tags all captured sensor data, and wherein said data conversion processing means automatically combinines specific items of sensor data into specific records having specific voice data and specific location data by using said time tagging means as a synchronizing key.

15. The invention defined in claim 12 wherein said location data capturing means comprises a mobile computer configured with a global positioning satellite receiver to capture the location data.

16. The invention defined in claim 12 wherein said free speech capturing means comprises a mobile computer configured with a sound board for capturing said free speech stating said verbal observation containing the voice data.

17. The invention defined in claim 12 wherein said free speech capturing means and said location data capturing means comprise a mobile computer for capturing both the location data and said verbal observation containing the voice data and including a transfer means for transferring the captured location data and the captured voice data from said mobile computer to a processing computer which combines the captured voice data and the captured location data into said single record which is stored in said geographic information system data base associated with said processing computer.

18. The invention defined in claim 12 wherein said reference grammar means and said semantic relationship structure means are configured to a specific form for a specific application.

19. The invention defined in claim 17 wherein said specific application is a street maintenance application and said reference grammar means and said semantic relationship structure means are configured to a specific form for use in the specific street maintenance application.

20. The invention defined in claim 12 wherein said free speech capturing means continuously captures all audio input and wherein said data conversion processing means automatically removes all non free speech audio input.

21. The invention defined in claim 12 including a correlation means for automated correlation of the captured location data with the captured voice data.

22. The invention defined in claim 21 wherein said correlation means provides automated correlation of the location data to linear networks.

23. The invention defined in claim 21 wherein said correlation means provides automated correlation of the location data to point observations.

24. The invention defined in claim 21 wherein said correlation means provides automated correlation of the location data to said raw verbal observation and then to certain layers of said geographic information system data base.

25. A method of combining captured location data with captured voice data by using a predefined reference grammar with a semantic relationship structure, said method comprising:

constructing the predefined reference grammar for the voice data to be captured;

incorporating the semantic relationship structure in the predefined reference grammar;

capturing and time tagging a first free speech stating a first verbal observation containing a first voice data to create a first raw verbal observation;

capturing and time tagging a second free speech stating a second verbal observation containing a second voice data to create a second raw verbal observation;

capturing a first location data contemporaneously with said first raw verbal observation and time tagging said captured first location data;

capturing a second location data contemporaneously with said second raw verbal observation and time tagging said captured second location data;

using the predefined reference grammar and the semantic relationship structure, determining if said first raw verbal observation and said second raw verbal observation are related; and

when said first raw verbal observation and said second raw verbal observation are related, combining said first raw verbal observation and said second raw verbal observation automatically with said captured first location data and said captured second location data into a single record in a geographic information system data base.

26. The invention defined in claim 25 wherein when said first raw verbal observation and said second raw verbal observation are not related, combining said first raw verbal observation with said captured first location data into a first record in said geographic information system data base, and combining said second raw verbal observation with said captured second location data into a second record in said geographic information system data base.

27. The invention defined in claim 25 including using a mobile computer in communication with a location receiver device for capturing the location data.

28. A spatial asset management system apparatus for combining captured location data with captured voice data by using a reference grammar with a semantic relationship structure, said apparatus comprising:

a reference grammar means for interpreting the voice data captured in a first verbal observation and a second verbal observation;

said reference grammar means incorporating a semantic relationship structure means for analyzing the captured voice data contained in said first and second verbal observations to determine if said first and second verbal observations are related;

free speech capturing means for capturing free speech stating said first and second verbal observations containing the voice data;

location data capturing means for capturing the location data contemporaneously with said first and second verbal observations;

time tagging means for time tagging the voice data captured in said first and second verbal observations to create a first raw verbal observation and a second raw verbal observation and for also time tagging the captured location data for said first and second verbal observations; and

data conversion processing means for using said reference grammar means and said semantic relationship structure means to automatically combine said first and second raw verbal observations when they are related with the captured location data for said first and second raw verbal observations into a single record in a geographic information system data base.

29. The invention defined in claim 28 wherein when said first raw verbal observation and said second raw verbal observation are not related, said data conversion processing means combines said first raw verbal observation with said captured first location data into a first record in said geographic information system data base, and combines said second raw verbal observation with said captured second location data into a second record in said geographic information system data base.

30. The invention defined in claim 28 wherein said location data capturing means comprises a mobile computer in communication with a location receiver device for capturing the location data.