COMPUTER VISION AND MACHINE LEARNING SOFTWARE FOR GRADING AND SORTING PLANTS

• US 20130028487A1
• Filed: 03/14/2011
• Published: 01/31/2013
• Est. Priority Date: 03/13/2010
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: LaRose, David Arthur, Bagnell, James Andrew, Stager, David Jonathan, Sergi-Curfman, Michael Knoll, Fromme, Christopher Chandler, Cuzzillo, Elliot Allen, Baker, L. Douglas
• Granted Patent: US 9,527,115 B2
• Time in Patent Office: Days
• US Class Current: 382/110
• CPC Class Codes: B07C 5/342 according to optical properties, e.g. colour according to radiation transmittivity B07C5/3416

Abstract

The present invention encompasses software that brings together computer vision and machine learning algorithms that can evaluate and sort plants into desired categories. While one embodiment of the present invention is directed toward strawberry plants, the software engine described is not specifically designed for strawberry plants but can be used for many different types of plants that require sophisticated quality sorting. The present invention is a sequence of software operations that can be applied to various crops (or other objects besides plants) in a re-usable fashion.

Claims

1. A system to recognize and classify a bare-root plant on a surface, the system comprising the steps of:

classifying a bare-root plant from a raw image to form a classified bare-root image based on trained parameters; and

evaluating the classified bare-root plant based on trained features to assign the evaluated bare-root plant to a configured category,

whereby, the bare-root plant can be dispositioned based on the assigned configured category.

2. The system according to claim 1, wherein the step of classifying comprises a step of detecting and extracting foreground objects to identify a plurality of sub-parts of the bare-root plant to form a cropped image.

3. The system according to claim 2, wherein the step of detecting and extracting foreground objects comprises a step of creating the cropped image containing only the bare-root plant and minimal background.

4. The system according to claim 2, wherein the step of detecting and extracting objects comprises a step of color masking.

5. The system according to claim 4, wherein the step of color masking comprises a step of selecting a color of the surface to form a background color being maximally different compared to colors of the plurality of sub-parts of the bare-root plant to form foreground color.

6. The system according to claim 5, wherein the step of selecting the color of the surface to form the background color comprises the step of selecting the background color that is out of phase with the foreground color.

7. The system according to claim 6, wherein the step of selecting the color of the surface to form the background color further comprises a step of segregating the foreground color and the background color based on a hue threshold to create a mask.

8. The system according to claim 7, wherein the step of color masking further comprises a step of applying the mask to the raw image to form the classified bare-root image, whereby only the foreground color is displayed.

9. The system according to claim 2, wherein the step of classifying the bare-root plant from the raw image to form the classified bare-root image further comprises a step of calculating features for use in pixel classification based on the cropped image to classify each pixel of the cropped image as one sub-part of the plurality of sub-parts of the bare-root plant to form a portion of a vector of scores.

10. The system according to claim 9, wherein the step of classifying the bare-root plant from the raw image to form the classified bare-root image further comprises a step of calculating the neighborhood mean and the variance for each pixel based on the calculated features to form another portion of the vector of scores.

11. The system according to claim 9, wherein the step of classifying the bare-root plant from the raw image to form the classified bare-root image further comprises a step of down-selecting to reduce the number of calculated features.

12. The system according to claim 9, wherein the step of classifying the bare-root plant from the raw image to form the classified bare-root image further comprises a step of classifying pixels based on the calculated features.

13. The system according to claim 12, wherein the step of classifying pixels based on the calculated features comprises a step of calculating feature vector scores.

14. The system according to claim 13, wherein the step of calculating feature vector scores comprises a step of machine learning to associate feature vector scores with particular sub-parts of the plurality of sub-parts of the bare-root plant.

15. The system according to claim 2, further comprising a step of masking of disconnected components of the cropped image to remove foreground pixels that are not part of the bare-root plant.

16. The system according to claim 1, wherein the step of evaluating the classified bare-root plant based on trained features to assign the evaluated bare-root plant to a configured category comprises a step of calculating features of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image.

17. The system according to claim 16, wherein the step of calculating features of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image comprises a step of virtual cropping.

18. The system according to claim 16, wherein the step of evaluating the classified bare-root plant based on trained features to assign the evaluated bare-root plant to a configured category further comprises a step of classifying the bare-root plant based on the vector of scores for each plant image.

19. The system according to claim 18, wherein the step of classifying the bare-root plant based on the vector of scores for each plant image comprises a step of machine learning to associate the vector of scores to assign the configured category to the bare-root plant.

20. The system according to claim 18, further comprising a step of calculating features for use in multiple plant detection based on the vector of scores.

21. The system according to claim 20, wherein the step of calculating features for use in multiple plant detection based on the vector of scores comprises a step of calculating vector of scores for multiple plant detection.

22. The system according to claim 21, step of calculating vector of scores for multiple plant detection further comprises a step of machine learning applied to the vector of scores for multiple plant detection to associate between a single bare-root plant and multiple bare-root plants.

23. The system according to claim 20, further comprising a step of detecting a single or multiple plants.

24. The system according to claim 14, wherein the step of machine learning comprises:

gathering examples of each category of bare-root plant;

acquiring an isolated image of each example;

establishing a training example based on foreground pixels from each isolated image; and

creating a model for a classifier based on the training example to build associations of the vector of scores to the plurality of sub-parts of the bare-root plant.

25. The system according to claim 19, wherein the step of machine learning comprises:

gathering examples of each category of the bare-root plant;

acquiring an isolated image of each example;

establishing a training example based on foreground pixels from each isolated image; and

creating a model for a classifier based on the training example to build associations of the vector of scores to plant categories.

26. The system according to claim 14, wherein the step of machine learning comprises:

gathering examples of images of complete bare-root plants;

processing each image with a super-pixel algorithm utilizing intensity and hue space segmentation;

labeling of the foreground pixels of each image for each sub-part of the complete bare-root plants to form a training example; and

creating a model for a classifier based on the training example to build associations of the vector of scores to the plurality of sub-parts of the bare-root plant.

27. A method for sort and grade plants comprising the steps of:

detecting and extracting of foreground objects from a raw image of a plant;

calculating features from use in pixel classification from the raw image of the plant;

pixel classifying of plant sub-parts from the raw image of the plant; and

classifying the plant into plant categories based on training.

28. A system to recognize and classify a bare-root plant on a surface, the system comprising the steps of:

detecting and extracting foreground objects to identify a plurality of sub-parts of the bare-root plant to form a cropped image;

calculating features for use in pixel classification based on the cropped image to classify each pixel of the cropped image as one sub-part of the plurality of sub-parts of the bare-root plant;

classifying pixels of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image;

calculating features for use in plant classification; and

classifying the bare-root plant based on the calculated features into a configured category,

whereby, the bare-root plant is capable of being dispositioned based on the configured category.

29. A system to recognize and classify a bare-root plant on a surface, the system comprising the steps of:

detecting and extracting foreground objects to identify a plurality of sub-parts of the bare-root plant to form a first cropped image;

masking disconnected components of the first cropped image to form a second cropped image;

calculating features for use in pixel classification based on the second cropped image to classify each pixel of the cropped image as one sub-part of the plurality of sub-parts of the bare-root plant;

classifying pixels of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image;

calculating features for use in plant classification;

calculating features for use in multiple plant detection;

detecting a single plant or multiple plants; and

classifying the bare-root plant based on the calculated features into a configured category,

whereby, the bare-root plant is capable of being dispositioned based on the configured category.