Computer vision and machine learning software for grading and sorting plants

• US 9,527,115 B2
• Filed: 03/14/2011
• Issued: 12/27/2016
• Est. Priority Date: 03/13/2010
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: LaRose, David Arthur, Fromme, Christopher Chandler, Stager, David Jonathan, Sergi-Curfman, Michael Knoll, Bagnell, James Andrew, Cuzzillo, Elliot Allen, Baker, L. Douglas
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US13/634,086
• Publication Number:
  US 20130028487A1
• Time in Patent Office: 2,115 Days
• US Class Current: 1/1
• CPC Class Codes: B07C 5/342 according to optical properties, e.g. colour according to radiation transmittivity B07C5/3416

Abstract

The present invention encompasses software that brings together computer vision and machine learning algorithms that can evaluate and sort plants into desired categories. While one embodiment of the present invention is directed toward strawberry plants, the software engine described is not specifically designed for strawberry plants but can be used for many different types of plants that require sophisticated quality sorting. The present invention is a sequence of software operations that can be applied to various crops (or other objects besides plants) in a re-usable fashion.

Claims

1. A method to recognize and classify a bare-root plant on a surface, comprising the steps of:

receiving from an imaging device a continuous output of raw image data of several bare-root plants passing through a field of view of the imaging device,

wherein each of the several bare-root plants are arbitrarily disposed on the surface and positioned between the surface and the imaging device;

wherein the surface is horizontal and each of the several bare-root plants is lying flat on the surface; and

wherein an orientation of each of the several bare-root plants on the surface is not uniform;

identifying a single bare-root plant in the raw image data by detecting and extracting a region in the raw image data corresponding to the single plant;

classifying each pixel of the bare-root plant identified in the raw image data to form a classified bare-root image based on trained parameters, comprising;

generating a vector of scores for each pixel of the classified bare-root image; and

identifying a plurality of sub-parts of the bare-root plant based at least on the vector of scores and the trained parameters;

evaluating the classified bare-root image based on trained features to assign the bare-root plant to a configured category; and

sorting the bare-root plant based on the assigned configured category as it moves from the field of view, using a sorting device in communication with a vision system.

2. The method according to claim 1, wherein the step of classifying comprises a step of detecting and extracting foreground objects to form a cropped image.

3. The method according to claim 2, wherein the step of detecting and extracting foreground objects comprises a step of creating the cropped image containing only the bare-root plant and minimal background.

4. The method according to claim 2, wherein the step of detecting and extracting objects comprises a step of color masking.

5. The method according to claim 4, wherein the step of color masking comprises a step of selecting a color of the surface to form a background color being maximally different compared to colors of the plurality of sub-parts of the bare-root plant to form foreground color.

6. The method according to claim 5, wherein the step of selecting the color of the surface to form the background color comprises the step of selecting the background color that is out of phase with the foreground color.

7. The method according to claim 6, wherein the step of selecting the color of the surface to form the background color further comprises a step of segregating the foreground color and the background color based on a hue threshold to create a mask.

8. The method according to claim 7, wherein the step of color masking further comprises a step of applying the mask to the raw image data to form the classified bare-root image, whereby only the foreground color is displayed.

9. The method according to claim 1, wherein the step of classifying each pixel of the bare-root plant identified in the raw image data to form the classified bare-root image further comprises a step of calculating the neighborhood mean and the variance for each pixel based on calculated features to form a portion of the vector of scores.

10. The method according to claim 9, wherein the step of classifying each pixel of the bare-root plant identified in the raw image data to form the classified bare-root image further comprises a step of down-selecting to reduce the number of calculated features.

11. The method according to claim 9, wherein the step of classifying each pixel of the bare-root plant identified in the raw image data to form the classified bare-root image further comprises a step of classifying each pixel of the bare-root plant based on the calculated features.

12. The method according to claim 1, wherein the step of generating the vector of scores comprises a step of machine learning to associate the vector of scores with particular sub-parts of the plurality of sub-parts of the bare-root plant.

13. The method according to claim 2, further comprising a step of masking of disconnected components of the cropped image to remove foreground pixels that are not part of the bare-root plant.

14. The method according to claim 1, wherein the step of evaluating the classified bare-root image based on trained features to assign the bare-root plant to a configured category comprises a step of calculating features of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each bare-root plant image.

15. The method according to claim 14, wherein the step of calculating features of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each bare-root plant image comprises a step of virtual cropping.

16. The method according to claim 14, wherein the step of evaluating the classified bare-root plant image based on trained features to assign the bare-root plant to a configured category further comprises a step of classifying the bare-root plant based on the vector of scores for each bare-root plant image.

17. The method according to claim 16, wherein the step of classifying the bare-root plant based on the vector of scores for each bare-root plant image comprises a step of machine learning to associate the vector of scores to assign the configured category to the bare-root plant.

18. The method according to claim 16, further comprising a step of calculating features for use in multiple plant detection based on the vector of scores.

19. The method according to claim 18, wherein the step of calculating features for use in multiple plant detection based on the vector of scores comprises a step of calculating vector of scores for multiple plant detection.

20. The method according to claim 19, wherein the step of calculating vector of scores for multiple plant detection further comprises a step of machine learning applied to the vector of scores for multiple plant detection to associate between a single bare-root plant and multiple bare-root plants.

21. The method according to claim 18, further comprising a step of detecting a single or multiple plants.

22. The method according to claim 12, wherein the step of machine learning comprises:

gathering examples of each category of bare-root plant;

acquiring an isolated image of each example;

establishing a training example based on foreground pixels from each isolated image; and

creating a model for a classifier based on the training example to build associations of the vector of scores to the plurality of sub-parts of the bare-root plant.

23. The method according to claim 17, wherein the step of machine learning comprises:

gathering examples of each category of the bare-root plant;

acquiring an isolated image of each example;

establishing a training example based on foreground pixels from each isolated image; and

creating a model for a classifier based on the training example to build associations of the vector of scores to plant categories.

24. The method according to claim 12, wherein the step of machine learning comprises:

gathering examples of images of complete bare-root plants;

processing each image with a super-pixel algorithm utilizing intensity and hue space segmentation;

labeling of the foreground pixels of each image for each sub-part of the complete bare-root plants to form a training example; and

creating a model for a classifier based on the training example to build associations of the vector of scores to the plurality of sub-parts of the bare-root plant.

27. The method of claim 1, wherein identifying a single bare-root plant in the raw image data further comprises:

comparing a count of foreground pixels along a first axis to a threshold count,

wherein the threshold count is based on a minimum size of plant to be detected,

wherein a plant part is detected in the foreground pixels if the count is higher than the threshold count; and

comparing a second count of foreground pixels along a second axis to the threshold count,

wherein the plant part is detected if the count is higher than the threshold count, thereby marking the plant part along the first axis and the second axis.

28. The method of claim 1, wherein identifying a single bare-root plant in the raw image data further comprises:

joining a first group of foreground pixels with an adjacent second set of foreground pixels if the first set and second set are connected; and

discarding the joined group of foreground pixels if the joined group is below a threshold size, thereby removing dirt and debris from the raw image data.

25. A method to recognize and classify a bare-root plant on a surface, comprising the steps of:

obtaining a raw image with an imaging device;

identifying a single bare-root plant in raw image by detecting and extracting a region in the raw image corresponding to the single bare-root plant, wherein the raw image contains several bare-root plants, wherein each of the several bare-root plants arbitrarily disposed on a surface and are positioned between the surface and the imaging device, wherein the surface is horizontal and each of the several bare-root plants is lying flat on the surface; and

wherein an orientation of each of the several bare-root plants on the surface is not uniform;

detecting and extracting foreground objects to identify a plurality of sub-parts of the bare-root plant to form a cropped image;

calculating features for use in pixel classification based on the cropped image to classify each pixel of the cropped image as one sub-part of the plurality of sub-parts of the bare-root plant;

classifying pixels of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image;

calculating category features for use in plant classification; and

classifying the bare-root plant based on the calculated category features into a configured category,

sorting, the bare-root plant based on the configured category as it moves from the field of view, using a sorting device in communication with a vision system.

26. A method to recognize and classify a bare-root plant on a surface, comprising the steps of:

identifying a single bare-root plant in a raw image by detecting and extracting a region in the raw image corresponding to the single bare-root plant, wherein the raw image contains several bare-root plants, each of the several bare-root plants arbitrarily disposed on a surface;

wherein the surface is horizontal and each of the several bare-root plants is lying flat on the surface; and

wherein an orientation of each of the several bare-root plants on the surface is not uniform;

detecting and extracting foreground objects to identify a plurality of sub-parts of the bare-root plant to form a first cropped image;

masking disconnected components of the first cropped image to form a second cropped image, wherein the masking step comprises;

joining a first group of foreground pixels with an adjacent group of foreground pixels, and

identifying the joined group of foreground pixels as disconnected components if a size of the joined group is less than a minimal value;

calculating features for use in pixel classification based on the second cropped image to classify each pixel of the cropped image as one sub-part of the plurality of sub-parts of the bare-root plant;

classifying pixels of the plurality of sub-parts of the bare-root plant to generate a vector of scores for each plant image;

calculating first category features for use in plant classification;

calculating second category features for use in multiple plant detection;

detecting a single plant or multiple plants; and

classifying the bare-root plant based on the calculated first category features into a configured category,

sorting, the bare-root plant based on the configured category as it moves from the field of view, using a sorting device in communication with a vision system.