Methods and systems for pattern characteristic detection

• US 10,747,999 B2
• Filed: 10/16/2018
• Issued: 08/18/2020
• Est. Priority Date: 10/18/2017
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Cornell University
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: DeChant, Alan Chad, Lipson, Hod, Nelson, Rebecca J., Gore, Michael A., Wiesner-Hanks, Tyr, Stewart, Ethan, Yosinski, Jason, Chen, Siyuan
• Primary Examiner(s): Patel, Mahendra R
• Application Number:
  US16/161,499
• Publication Number:
  US 20190114481A1
• Time in Patent Office: 672 Days
• Field of Search: 382110, 382157, 382228, 382133, 1 1, 706 48
• CPC Class Codes: A01B 79/005 Precision agriculture
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/188 Vegetation
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Disclosed are devices, systems, apparatus, methods, products, and other implementations, including a method to detect pattern characteristics in target specimens that includes acquiring sensor data for the target specimens, dividing the acquired sensor data into a plurality of data segments, and generating, by multiple neural networks that each receives the plurality of data segments, multiple respective output matrices, with each data element of the multiple respective output matrices being representative of a probability that corresponding sensor data of a respective one of the plurality of data segments includes a pattern characteristic in the target specimens. The method further includes determining by another neural network, based on the multiple respective output matrices generated by the multiple neural networks, a presence of the pattern characteristic in the target specimens.

Claims

1. A method to detect pattern characteristics in target specimens, the method comprising:

acquiring sensor data for the target specimens;

dividing the acquired sensor data into a plurality of data segments;

generating, by multiple neural networks that each receives the plurality of data segments, multiple respective output matrices, with each data element of the multiple respective output matrices being representative of a probability that corresponding sensor data of a respective one of the plurality of data segments includes a pattern characteristic in the target specimens; and

determining by another neural network, based on the multiple respective output matrices generated by the multiple neural networks, a presence of the pattern characteristic in the target specimens;

wherein the method further comprises;

providing training image data to train the multiple neural networks and the other neural network to detect northern leaf blight (NLB) disease in corn crops; and

training the multiple neural networks and the other neural network using the training image data, including;

identifying a lesion with a lesion axis in at least one image of the corn crops from the image data;

defining multiple image segments of predetermined dimensions that are each shifted, from another of the multiple image segments, by a predetermined length of pixels and having a center at a randomly selected location within a predetermined radius of pixels from the lesion axis of the lesion; and

rotating the each of the defined multiple image segments by a random rotation angle.

2. The method of claim 1, wherein the pattern characteristic is indicative of an abnormality of the target specimens.

3. The method of claim 1, wherein acquiring sensor data for the target specimens comprises:

acquiring image data for crop objects.

4. The method of claim 3, wherein acquiring the image data for the crop objects comprises:

capturing one or more of;

an aerial image of the crop objects, or a ground-based image of the crop objects.

5. The method of claim 3, wherein the image data for the crop objects comprises one or more images of the corn crops, and wherein the presence of the pattern characteristic in the corn crops is indicative of the presence of the northern leaf blight (NLB) disease in the corn crops.

6. The method of claim 1, wherein generating the multiple respective output matrices comprises:

providing the plurality of data segments to a plurality of convolutional neural network (CNN) units;

determining for a particular data segment, from the plurality of data segments, multiple probability values generated respectively by each of the plurality of CNN units; and

writing at a particular location, corresponding to the particular data segment, in each of the multiple respective output matrices, respective ones of the multiple probability values determined for the particular data segment.

7. The method of claim 1, wherein the acquiring, dividing, generating, and determining are performed in real-time.

8. The method of claim 1, wherein at least some of the dividing, generating, and determining are performed offline.

9. The method of claim 1, wherein determining by the other neural network the presence of the pattern characteristic in the target specimens comprises:

determining by the other neural network the presence of the pattern characteristic in the target specimens further based on at least a portion of the acquired sensor data.

10. The method of claim 1, wherein acquiring the sensor data for the target specimens comprises:

acquiring for the target specimens one or more of;

visible range optical data, non-visible range optical data, RF data, or environmental data.

11. The method of claim 1, wherein determining the presence of the pattern characteristic in the target specimens comprises:

determining a percentage measure of the pattern characteristic in the target specimens.

12. A detection system to detect pattern characteristics in target specimens, the system comprising:

one or more sensors to acquire sensor data for the target specimens;

a controller to divide the acquired sensor data into a plurality of data segments;

multiple neural networks, each configured to receive the plurality of data segments and to generate multiple respective output matrices, with each data element of the multiple respective output matrices being representative of a probability that corresponding sensor data of a respective one of the plurality of data segments includes a pattern characteristic in the target specimens; and

at least one other neural network configured to determine, based on the multiple respective output matrices generated by the multiple neural networks, a presence of the pattern characteristic in the target specimens;

wherein the controller is further configured to;

provide training image data to train the multiple neural networks and the other neural network to detect northern leaf blight (NLB) disease in corn crops; and

train the multiple neural networks and the other neural network using the training image data, including to;

identify a lesion with a lesion axis in at least one image, of the corn crops, from the image data;

define multiple image segments, from the at least one image, of predetermined dimensions that are each shifted, from another of the multiple image segments, by a predetermined length of pixels and having a center at a randomly selected location within a predetermined radius of pixels from the lesion axis of the lesion; and

rotate the each of the defined multiple image segments by a random rotation angle.

13. The detection system of claim 12, wherein the one or more sensors comprise at least one light-capture device to acquire image data for crop objects, wherein the image data comprises one or more of:

an aerial image of the crop objects, or a ground-based image of the crop objects.

14. The detection system of claim 13, wherein the image data for the crop objects comprises one or more images of the corn crops, and wherein the presence of the pattern characteristic in the corn crops is indicative of the presence of the northern leaf blight (NLB) disease in the corn corps.

15. The detection system of claim 12, wherein the multiple neural networks configured to generate multiple respective output matrices are each configured to:

provide the plurality of data segments to a plurality of convolutional neural network (CNN) units;

determine for a particular data segment, from the plurality of data segments, multiple probability values generated respectively by each of the plurality of CNN units; and

write at a particular location, corresponding to the particular data segment, in each of the multiple respective output matrices, respective ones of the multiple probability values determined for the particular data segment.

16. The detection system of claim 12, wherein the at least one other neural network configured to determine the presence of the pattern characteristic in the target specimens is configured to:

determine the presence of the pattern characteristic in the target specimens further based on at least a portion of the acquired sensor data.

17. The detection system of claim 12, wherein at least one of the one or more sensors comprises one or more of:

a visible range optical sensor, a non-visible range optical sensor, a RF receiver, or a sensor to measure one or more environmental characteristics.

18. A non-transitory computer readable media storing a set of instructions, executable on at least one programmable device, to:

acquire sensor data for target specimens;

divide the acquired sensor data into a plurality of data segments;

generate, by multiple neural networks that each receives the plurality of data segments, multiple respective output matrices, with each data element of the multiple respective output matrices being representative of a probability that corresponding sensor data of a respective one of the plurality of data segments includes a pattern characteristic in the target specimen; and

determine by another neural network, based on the multiple respective heatmaps generated by the multiple neural networks, a presence of the abnormality in the target specimens;

wherein the set of instructions comprises one or more further instructions, executable on the at least one programmable device, to further;

provide training image data to train the multiple neural networks and the other neural network to detect northern leaf blight (NLB) disease in corn crops; and

train the multiple neural networks and the other neural network using the training image data, including to;

identify a lesion with a lesion axis in at least one image, of the corn crops, from the image data;

define multiple image segments, from the at least one image, of predetermined dimensions that are each shifted, from another of the multiple image segments, by a predetermined length of pixels and having a center at a randomly selected location within a predetermined radius of pixels from the lesion axis of the lesion; and

rotate the each of the defined multiple image segments by a random rotation angle.