Systems, devices, and methods for robotic remote sensing for precision agriculture

• US 10,395,115 B2
• Filed: 01/27/2016
• Issued: 08/27/2019
• Est. Priority Date: 01/27/2015
• Status: Active Grant

• Current Assignee: Trustees Of The University Of Pennsylvania (University of Pennsylvania)
• Original Assignee: Trustees Of The University Of Pennsylvania (University of Pennsylvania)
• Inventors: Kumar, R. Vijay, Cross, Gareth Benoit, Qu, Chao, Das, Jnaneshwar, Makineni, Anurag, Mulgaonkar, Yash Shailesh
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/545,266
• Publication Number:
  US 20170372137A1
• Time in Patent Office: 1,308 Days
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06V 20/188 Vegetation
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: Y02A 40/10 in agriculture

Abstract

The present subject matter relates to systems, devices, and methods for data-driven precision agriculture through close-range remote sensing with a versatile imaging system. This imaging system can be deployed onboard low-flying unmanned aerial vehicles (UAVs) and/or carried by human scouts. Additionally, the present technology stack can include methods for extracting actionable intelligence from the rich datasets acquired by the imaging system, as well as visualization techniques for efficient analysis of the derived data products. In this way, the present systems and methods can help specialty crop growers reduce costs, save resources, and optimize crop yield.

Claims

1. A method for data-driven remote sensing for precision agriculture, the method comprising:

obtaining high-resolution 3-D imagery using one or more mobile sensor arrays;

applying state-estimation and mapping algorithms to the high-resolution 3-D imagery to generate high-resolution multi-spectral 3-D maps; and

extracting actionable intelligence from the high-resolution multi-spectral 3-D maps.

2. The method of claim 1, wherein obtaining high-resolution 3-D imagery comprises obtaining both side views and aerial views of crops in a subject agricultural area.

3. The method of claim 1, wherein at least one of the one or more mobile sensor arrays are deployed on a low-flying unmanned aerial vehicle (UAV).

4. The method of claim 1, wherein at least one of the one or more mobile sensor arrays are carried by human scouts.

5. The method of claim 1, wherein obtaining the high-resolution 3-D imagery comprises obtaining multi-spectral 3-D data.

6. The method of claim 1, wherein obtaining the high-resolution 3-D imagery comprises simultaneously exploring an unknown environment, determining the locations of the plurality of mobile sensor arrays with respect to the environment, and building a map of the environment.

7. The method of claim 1, wherein extracting the actionable intelligence comprises applying statistical models to predict properties of interest such as crop yield, trunk size, canopy volume, water stress, and/or disease.

8. A system for performing data-driven remote sensing for precision agriculture comprising:

a mobile deployment device;

a multispectral 3-D imaging system coupled to the mobile deployment device, the multispectral 3-D imaging system comprising a science sensor array and a navigation sensor array, the imaging system being configured to obtain high-resolution 3-D imagery of a subject agricultural area; and

a data visualization framework in communication with the science sensor array and the navigation sensor array, the data visualization framework being configured to apply state-estimation and mapping algorithms to the high-resolution 3-D imagery to generate high-resolution multi-spectral 3-D maps of the subject agricultural area.

9. The system of claim 8, wherein the science sensor array comprises one or more of a laser range scanner (LiDAR), one or more multi-spectral cameras spanning red and near-infrared bands, a thermal camera, and/or a spectrometer.

10. The system of claim 8, wherein the navigation sensor array comprises one or more stereo camera for visual odometry, a global positioning system (GPS) sensor or other navigational sensors, and/or an inertial measurement unit (IMU).

11. The system of claim 8, wherein the multispectral 3-D imaging system weighs less than 1.6 kg.

12. The system of claim 8, wherein the mobile deployment device comprises a low-flying unmanned aerial vehicle (UAV).

13. The system of claim 12, wherein the mobile deployment device comprises a plurality of low-flying UAVs operable in a swarming arrangement to collectively obtain the high-resolution 3-D imagery of the subject agricultural area.

14. The system of claim 8, wherein the mobile deployment device comprises a harness configured to be carried by human scouts.

15. The system of claim 8, wherein the data visualization framework comprises a comprehensive real-time and/or offline data visualization framework configured for efficient exploratory analysis of raw sensor data and derived data products obtained by the multispectral 3-D imaging system.

16. The system of claim 15, wherein the data visualization framework comprises:

a state-estimator configured to estimate the pose of the multispectral 3-D imaging system based on a navigation data stream generated by the navigation sensor array during motion;

a point-cloud assembler configured to generate a multi-spectral 3-D point cloud from the pose of the multispectral 3-D imaging system and a science data stream generated by the science sensor array; and

a machine-learning model configured for extracting actionable intelligence from the multi-spectral 3-D point cloud to generate the high-resolution multi-spectral 3-D maps of the subject agricultural area.

17. The system of claim 8, wherein the data visualization framework is located remotely from the multispectral 3-D imaging system;

and

wherein the multispectral 3-D imaging system comprises a wireless communications link configured for communication with the data visualization framework.