System and method to operate a drone

• US 10,095,243 B2
• Filed: 08/09/2016
• Issued: 10/09/2018
• Est. Priority Date: 12/08/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boland, Gregory F., Kozloski, James R., Ma, Yu, Manweiler, Justin G., Siemonsen, Kevin E., Topkara, Umut, Vogt, Katherine, Weisz, Justin D.
• Primary Examiner(s): CHEN, SHELLEY
• Application Number:
  US15/232,366
• Publication Number:
  US 20170160752A1
• Time in Patent Office: 791 Days
• Field of Search: 701 3
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/104 involving a plurality of aircrafts, e.g. formation flying traffic control systems for aircraft G08G5/00
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

A method for controlling a drone includes receiving a natural language request for information about a spatial location, parsing the natural language request into data requests, configuring a flight plan and controlling one or more drones to fly over the spatial location to obtain data types based on the data requests, and extracting and analyzing data to answer the request. The method can include extracting data points from the data types, obtaining labels from a user for one or more of the data points, predicting labels for unlabeled data points from a learning algorithm using the labels obtained from the user, determining the predicted labels are true labels for the unlabeled data points and combining the extracted data, the user labeled data points and the true labeled data points to answer the request for information. The learning algorithm may be active learning using a support vector machine.

Claims

1. A method for controlling one or more drones to respond to a request for information, comprising;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests, each data request of the plurality of data requests corresponding to a portion of data necessary to answer the natural language request;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests, each data type providing a corresponding portion of the data necessary to answer the natural language request;

extracting data responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones; and

analyzing the responsive data to provide an answer to the natural language request for information.

2. The method of claim 1, wherein the data types include one or more of data obtained from an imaging system and data obtained from one or more sensors and wherein the plurality of data requests include one or more of a set of data to be collected, a location from which the data set is to be collected, analytics to be performed on the data set a timeframe to collect the data set.

3. The method of claim 1, further comprising selecting one or more drones based on matching drone capabilities to one or more of the plurality of data requests and uploading flight plans to the one or more drones, receiving real-time telemetry from the drone and performing analytics on the real-time telemetry to determine real-time flight conditions.

4. The method of claim 3, further comprising displaying the real-time telemetry and real-time flight conditions on a user interface (UI) in a mobile application and manually controlling the flight path of the one or more drones from the UI.

5. The method of claim 1, further comprising:

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

obtaining labels for one or more of the plurality of data points; and

combining the extracted data and the labeled data points to provide an answer to the first natural language request for information.

6. The method of claim 5, further comprising:

obtaining labels from a user for one or more of the plurality of data points;

predicting labels for unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points; and

combining the extracted data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

7. The method of claim 1, further comprising;

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

wherein configuring a flight plan comprises configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

wherein controlling one or more drones comprises controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

wherein extracting data comprises extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points; and

wherein analyzing responsive data comprises combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

8. The method of claim 7, wherein the learning algorithm comprises active learning using a support vector machine.

9. The method of claim 8, further comprising:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user label or a predicted true label for each of the plurality of data points; and

combining the second data, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.

10. A non-transitory article of manufacture tangibly embodying computer readable instructions, which when implemented, cause a computer to perform the steps of a method for controlling one or more drones to respond to a request for information, comprising;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests, each data request of the plurality of data requests corresponding to a portion of data necessary to answer the natural language request;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests, each data type providing a corresponding portion of the data necessary to answer the natural language request;

extracting data responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones; and

analyzing the responsive data to provide an answer to the natural language request for information.

11. The non-transitory article of manufacture of claim 10, further comprising computer readable instructions, which when implemented, cause a computer to perform the steps of:

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones, wherein the data types include one or more of data obtained from an imaging system and data obtained from one or more sensors and wherein the plurality of data requests include one or more of a set of data to be collected, a location from which the data set is to be collected, analytics to be performed on the data set a timeframe to collect the data set;

obtaining labels for one or more of the plurality of data points; and

combining the extracted data and the labeled data points to provide an answer to the first natural language request for information.

12. The non-transitory article of manufacture of claim 11, further comprising computer readable instructions, which when implemented, cause a computer to perform the steps of:

obtaining labels from a user for one or more of the plurality of data points;

predicting labels for unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points; and

combining the extracted data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

13. The non-transitory article of manufacture of claim 10, further comprising computer readable instructions, which when implemented, cause a computer to perform the steps of:

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

wherein configuring a flight plan comprises configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

wherein controlling one or more drones comprises controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

wherein extracting data comprises extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones

obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user, the learning algorithm comprising active learning using a support vector machine;

determining the predicted labels are true labels for the unlabeled data points; and

wherein analyzing the responsive data comprises combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

14. The non-transitory article of manufacture of claim 13, further comprising computer readable instructions, which when implemented, cause a computer to perform the steps of:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user label or a predicted true labels for each of the plurality of data points; and

combining the existing source obtained data points, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.

15. A computer system for controlling one or more drones to respond to a request for information, comprising:

one or more computer processors;

one or more non-transitory computer-readable storage media;

program instructions, stored on the one or more non-transitory computer-readable storage media, which when implemented by the one or more processors, cause the computer system to perform the steps of;

receiving a natural language request for information about a spatial location;

parsing the natural language request into a plurality of data requests, each data request of the plurality of data requests corresponding to a portion of data necessary to answer the natural language request;

configuring a flight plan for one or more drones over the spatial location based on the plurality of data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the plurality of data requests, each data type providing a corresponding portion of the data necessary to answer the natural language request;

extracting data responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones; and

analyzing the responsive data to provide an answer to the natural language request for information.

16. The computer system of claim 15, further comprising program instructions which cause the computer system to perform the steps of:

uploading flight plans to the one or more drones;

receiving real-time telemetry from the drone;

performing analytics on the real-time telemetry to determine real-time flight conditions, and

displaying the real-time telemetry and real-time flight conditions on a user interface (UI) in a mobile application.

17. The computer system of claim 15, further comprising program instructions which cause the computer system to perform the steps of:

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones, wherein the data types include one or more of data obtained from an imaging system and data obtained from one or more sensors and wherein the plurality of data requests include one or more of a set of data to be collected, a location from which the data set is to be collected, analytics to be performed on the data set a timeframe to collect the data set;

obtaining labels for one or more of the plurality of data points; and

combining the extracted data and the labeled data points to provide an answer to the first natural language request for information.

18. The computer system of claim 17, further comprising program instructions which cause the computer system to perform the steps of:

obtaining labels from a user for one or more of the plurality of data points;

predicting labels for unlabeled data points from a learning algorithm using the labels obtained from the user;

determining the predicted labels are true labels for the unlabeled data points; and

combining the extracted data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

19. The computer system of claim 15, further comprising program instructions which cause the computer system to perform the steps of:

searching for existing sources for the plurality of data requests;

determining that there are one or more existing sources for one or more of the plurality of data requests;

analyzing the existing sources to obtain first data responsive to the plurality of data requests;

determining that there are no existing sources for two or more of the plurality of data requests and identifying the data requests with no existing source as missing data requests;

wherein configuring a flight plan comprises configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

wherein controlling one or more drones comprises controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

wherein extracting data comprises extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

obtaining labels from a user for one or more of the plurality of data points;

determining whether there are unlabeled data points;

predicting labels for the unlabeled data points from a learning algorithm using the labels obtained from the user, the learning algorithm comprising active learning using a support vector machine;

determining the predicted labels are true labels for the unlabeled data points; and

wherein analyzing the responsive data comprises combining the first data, the user labeled data points and the true labeled data points to provide an answer to the first natural language request for information.

20. The computer system of claim 19, further comprising program instructions which cause the computer system to perform the steps of:

receiving a second natural language request for information about the spatial location;

parsing the second natural language request into a plurality of second data requests;

searching for existing sources for the plurality of second data requests;

determining that there are one or more existing sources for one or more of the plurality of second data requests;

analyzing the existing sources to obtain second data responsive to the plurality of second data requests;

determining that there are no existing sources for two or more of the plurality of second data requests and identifying the data requests with no existing source as missing data requests;

configuring a flight plan for one or more drones over the spatial location based on the missing data requests;

controlling one or more drones to fly over the spatial location according to the configured flight plan to obtain a plurality of data types from the spatial location based on the missing data requests;

extracting a plurality of data points responsive to the plurality of data requests from the plurality of data types obtained by the one or more drones;

determining that there is a user labels or predicted true label for each of the plurality of data points; and

combining the existing source obtained data points, the user labeled data points and the predicted true labeled data points to provide an answer to the second natural language request for information.