System and method to operate a drone

• US 9,471,064 B1
• Filed: 12/08/2015
• Issued: 10/18/2016
• Est. Priority Date: 12/08/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boland, Gregory F., Ma, Yu, Manweiler, Justin G., Siemonsen, Kevin E., Topkara, Umut, Vogt, Katherine, Kozloski, James R., Weisz, Justin D.
• Primary Examiner(s): Chen, Shelley
• Application Number:
  US14/962,147
• Time in Patent Office: 315 Days
• Field of Search: 701/3
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/104 involving a plurality of aircrafts, e.g. formation flying traffic control systems for aircraft G08G5/00
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

A method for controlling a drone includes receiving a natural language request for information about a spatial location, parsing the natural language request into data requests, configuring a flight plan and controlling one or more drones to fly over the spatial location to obtain data types based on the data requests, and extracting and analyzing data to answer the request. The method can include extracting data points from the data types, obtaining labels from a user for one or more of the data points, predicting labels for unlabeled data points from a learning algorithm using the labels obtained from the user, determining the predicted labels are true labels for the unlabeled data points and combining the extracted data, the user labeled data points and the true labeled data points to answer the request for information. The learning algorithm may be active learning using a support vector machine.

Claims

1. A method for controlling one or more drones to respond to a request for information, comprising;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests;

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests and based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests and based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points;

analyzing the responsive data to provide an answer to the natural language request for information, the analyzing including combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

2. The method of claim 1, wherein the data types includes one or more of data obtained from an imaging system and data obtained from one or more sensors.

3. The method of claim 1, wherein configuring a flight plan for one or more drones comprises selecting one or more drones based on matching drone capabilities to one or more of the plurality of data requests.

4. The method of claim 1, wherein controlling the one or more drones includes one or more of uploading flight plans to the one or more drones, receiving real-time telemetry from the drone and performing analytics on the real-time telemetry to determine real-time flight conditions.

5. The method of claim 4, wherein controlling the one or more drones includes displaying the real-time telemetry and real-time flight conditions on a user interface (UI) in a mobile application.

6. The method of claim 5, further comprising manually controlling the flight path of the one or more drones from the UI.

7. The method of claim 1, further comprising storing the plurality of data types obtained by the one or more drones and creating location and time indices for recall of the plurality OF Data types.

8. The method of claim 1, wherein the learning algorithm comprises active learning using a support vector machine.

9. The method of claim 8, further comprising:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user label or a predicted true label for each of the plurality of data points; and

combining the second data, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.

10. A non-transitory article of manufacture tangibly embodying computer readable instructions, which when implemented, cause a computer to perform the steps of a method for controlling one or more drones to respond to a request for information, comprising;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests;

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests and based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests and based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points;

analyzing the responsive data to provide an answer to the natural language request for information, the analyzing including combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

11. The non-transitory article of manufacture of claim 10, wherein controlling the one or more drones comprises one or more of uploading flight plans to the one or more drones, receiving real-time telemetry from the drone and performing analytics on the real-time telemetry to determine real-time flight conditions.

12. The non-transitory article of manufacture of claim 10, wherein controlling the one or more drones comprises displaying the real-time telemetry and real-time flight conditions on a user interface (UI) in a mobile application.

13. The non-transitory article of manufacture of claim 10, further comprising computer readable instructions, which when implemented, cause a computer to perform the steps of:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user label or a predicted true labels for each of the plurality of data points; and

combining the existing source obtained data points, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.

14. A computer system for controlling one or more drones to respond to a request for information, comprising:

one or more computer processors;

one or more non-transitory computer-readable storage media;

program instructions, stored on the one or more non-transitory computer-readable storage media, which when implemented by the one or more processors, cause the computer system to perform the steps of;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests;

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests and based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests and based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points;

analyzing the responsive data to provide an answer to the natural language request for information, the analyzing including combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

15. The computer system of claim 14, wherein controlling the one or more drones includes one or more of uploading flight plans to the one or more drones, receiving real-time telemetry from the drone and performing analytics on the real-time telemetry to determine real-time flight conditions.

16. The computer system of claim 14, wherein controlling the one or more drones includes displaying the real-time telemetry and real-time flight conditions on a user interface (UI) in a mobile application.

17. The computer system of claim 14, further comprising program instructions which cause the computer system to perform the steps of:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user labels or predicted true label for each of the plurality of data points; and

combining the existing source obtained data points, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.