DETECTION OF PLANT DISEASES WITH MULTI-STAGE, MULTI-SCALE DEEP LEARNING

• US 20200134392A1
• Filed: 10/23/2019
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Climate Pty., Ltd.
• Original Assignee: The Climate Corporation (Bayer AG)
• Inventors: GUI, YICHUAN, GUAN, WEI
• Granted Patent: US 10,713,542 B2
• Time in Patent Office: Days
• CPC Class Codes: A01B 79/005 Precision agriculture
• CPC Class Codes: A01G 7/00 Botany in general
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/24317 Piecewise classification, i.e. whereby each classification requires several discriminant rules
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/188 Vegetation
• CPC Class Codes: G06V 20/60 : Type of objects
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables

Abstract

In some embodiments, the system is programmed to build from multiple training sets multiple digital models, each for recognizing plant diseases having symptoms of similar sizes. Each digital model can be implemented with a deep learning architecture that classifies an image into one of several classes. For each training set, the system is thus programmed to collect images showing symptoms of one or more plant diseases having similar sizes. These images are then assigned to multiple disease classes. For a first one of the training sets used to build the first digital model, the system is programmed to also include images that correspond to a healthy condition and images of symptoms having other sizes. These images are then assigned to a no-disease class and a catch-all class. Given a new image from a user device, the system is programmed to then first apply the first digital model. For the portions of the new image that are classified into the catch-all class, the system is programmed to then apply another one of the digital models. The system is programmed to finally transmit classification data to the user device indicating how each portion of the new image is classified into a class corresponding to a plant disease or no plant disease.

Claims

1. A computer-implemented method of recognizing plant diseases having multi-sized symptoms from a plant image, comprising:

obtaining, by a processor, a first training set from at least a first photo showing a first symptom of one of a first plurality of plant diseases, a second photo showing no symptom, and a third photo showing a partial second symptom of one of a second plurality of plant diseases,

the first symptom being smaller than the second symptom,

the first, second, and third photos corresponding to similarly-sized fields of view;

building, by the processor, a first digital model from the first training set for classifying an image into a class of a first set of classes corresponding to the first plurality of plant diseases, a healthy condition, or a combination of the second plurality of plant diseases;

obtaining a second training set from at least a fourth photo showing the second symptom;

building a second digital model from the second training set for classifying an image into a class of a second set of classes corresponding to the second plurality of plant diseases;

receiving a new image from a user device;

applying the first digital model to a plurality of first regions within the new image to obtain a plurality of classifications;

applying the second digital model to one or more second regions, each corresponding to a combination of multiple first regions of the plurality of first regions, to obtain one or more classifications,

the multiple first regions being classified into the class corresponding to the combination of the second plurality of plant diseases;

transmitting classification data related to the plurality of classifications into a class corresponding to one of the first plurality of plant diseases or the healthy condition and the one or more classifications to the user device.

2. The computer implemented method of claim 1, the first digital model or the second digital model being a convolutional neural network (CNN) or a decision tree.

3. The computer implemented method of claim 1,

the first plurality of plant diseases including Common Rust, Eyespot, Southern Rust, or Gray Leaf Spot at an early stage,

the second plurality of plant diseases including Goss'"'"'s Wilt, Northern Leaf Blight, or Gray Leaf Spot at a late stage.

4. The computer implemented method of claim 1, the obtaining the first training set comprising:

identifying a size of a sliding window;

determining a first scaling factor;

determining a first image size based on the size of the sliding window and the first scaling factor;

resizing the first photo, the second photo, or the third photo according to the first image size to obtain a first resized photo, a second resize photo, or a third resized photo.

5. The computer implemented method of claim 4, the obtaining the second training set comprising:

determining a second scaling factor smaller than the first scaling factor;

determining a second image size based on the size of the sliding window and the second scaling factor;

resizing the fourth photo according to the second image size to obtain a fourth resized photo.

6. The computer implemented method of claim 5, further comprising:

determining a first stride and a second stride smaller than the first stride,

the obtaining the first training set further comprising extracting a first set of areas from the first resized photo, the second resized photo, or the third resized photo using the sliding window with the first stride,

the obtaining the second training set further comprising extracting a second set of areas from the fourth resized photo using the sliding window with the second stride.

7. The computer-implemented method of claim 6,

the obtaining the first training set further comprising assigning a label of a class of the first set of classes to each of the first set of areas,

the building the first digital model being performed further from the set of labels assigned to the first set of areas,

the obtaining the second training set further comprising assigning a label of a class of the second set of classes to each of the second set of areas,

the building the second digital model being performed further from the set of labels assigned to the second set of areas.

8. The computer-implemented method of claim 6, the applying the first digital model comprising:

resizing the new image according to the first image size to obtain a first updated image;

extracting the first plurality of regions from the first updated image using the sliding window with the first stride.

9. The computer-implemented method of claim 8, the applying the second digital model comprising:

masking each of the plurality of first regions in the new image that is classified into a class corresponding to one of the first plurality of plant diseases or a healthy condition to obtain a masked image;

resizing the masked image according to the second image size to obtain a second updated image;

extracting the one or more second regions from the second updated image using the sliding window with the second stride.

10. The computer-implemented method of claim 1, the applying the second digital model comprising resizing a portion of a combination of multiple first regions of the first plurality of regions to obtain the one second region.

11. The computer-implemented method of claim 1, the first training set being further obtained from a specific photo showing a third symptom of one of the first plurality of plant diseases and a fourth symptom of one of the second plurality of plant diseases, the fourth symptom overlapping with the third symptom.

12. The computer implemented method of claim 1, further comprising:

computing a total size of the plurality of first regions and the one or more second regions classified into each of the first set of classes and the second set of classes;

determining a dominant class of the first set of classes and the second set of classes such that the total size of the plurality of first regions and the one or more second regions classified into the dominant class is largest;

the classification data including information regarding the dominant class.

13. One or more non-transitory computer-readable media storing one or more sequences of instructions which when executed cause one or more processors to execute a method of recognizing plant diseases having multi-sized symptoms from a plant image, the method comprising:

obtaining a first training set from at least a first photo showing a first symptom of one of a first plurality of plant diseases, a second photo showing no symptom, and a third photo showing a partial second symptom of one of a second plurality of plant diseases,

the first symptom being smaller than the second symptom,

the first, second, and third photos corresponding to similarly-sized fields of view;

building a first digital model from the first training set for classifying an image into a class of a first set of classes corresponding to the first plurality of plant diseases, a healthy condition, or a combination of the second plurality of plant diseases;

obtaining a second training set from at least a fourth photo showing the second symptom;

building a second digital model from the second training set for classifying an image into a class of a second set of classes corresponding to the second plurality of plant diseases;

receiving a new image from a user device;

applying the first digital model to a plurality of first regions within the new image to obtain a plurality of classifications;

applying the second digital model to one or more second regions, each corresponding to a combination of multiple first regions of the plurality of first regions, to obtain one or more classifications,

the multiple first regions being classified into the class corresponding to the combination of the second plurality of plant diseases;

transmitting classification data related to the plurality of classifications into a class corresponding to one of the first plurality of plant diseases or the healthy condition and the one or more classifications to the user device.

14. The one or more non-transitory computer-readable media of claim 13, the obtaining a first training set comprising:

identifying a size of a sliding window;

determining a first scaling factor;

determining a first image size based on the size and the first scaling factor;

resizing the first photo, the second photo, or the third photo according to the first image size to obtain a first resized photo, a second resize photo, or a third resized photo.

15. The one or more non-transitory computer-readable media of claim 14, the obtaining a second training set comprising:

determining a second scaling factor smaller than the first scaling factor;

determining a second image size based on the size of the sliding window and the second scaling factor;

resizing the fourth photo according to the second image size to obtain a fourth resized photo.

16. The one or more non-transitory computer-readable media of claim 15, the method further comprising:

determining a first stride and a second stride smaller than the first stride,

the obtaining the first training set further comprising extracting a first set of areas from the first resized photo, the second resized photo, or the third resized photo using the sliding window with the first stride,

the obtaining the second training set further comprising extracting a second set of areas from the fourth resized photo using the sliding window with the second stride.

17. The one or more non-transitory computer-readable media of claim 16,

the obtaining the first training set further comprising assigning a label of a class of the first set of classes to each of the first set of areas,

the building the first digital model being performed further from the set of labels assigned to the first set of areas,

the obtaining the second training set further comprising assigning a label of a class of the second set of classes to each of the second set of areas,

the building the second digital model being performed further from the set of labels assigned to the second set of areas.

18. The one or more non-transitory computer-readable media of claim 16, the applying the first digital model comprising:

resizing the new image according to the first image size to obtain a first updated image;

extracting the first plurality of regions from the first updated image using the sliding window with the first stride.

19. The one or more non-transitory computer-readable media of claim 18, the applying the second digital model comprising:

masking each of the plurality of first regions in the new image that is classified into a class corresponding to one of the first plurality of plant diseases or a healthy condition to obtain a masked image;

resizing the masked image according to the second image size to obtain a second updated image;

extracting the one or more second regions from the second updated image using the sliding window with the second stride.

20. The one or more non-transitory computer-readable media of claim 13, the applying the second digital model comprising resizing a portion of a combination of multiple first regions of the first plurality of regions to obtain the one second region.