Monitoring and control implement for crop improvement

• US 10,721,859 B2
• Filed: 01/27/2019
• Issued: 07/28/2020
• Est. Priority Date: 01/08/2017
• Status: Active Grant

• Current Assignee: Dolly Y WU PLLC
• Original Assignee: Dolly Y WU PLLC
• Inventors: Wu, Dolly Y., Deutsch, Timothy A.
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US16/258,618
• Publication Number:
  US 20190150357A1
• Time in Patent Office: 548 Days
• Field of Search: 348 89
• CPC Class Codes: A01C 21/005 Following a specific plan, e.g. pattern
• CPC Class Codes: A01C 21/007 Determining fertilization requirements
• CPC Class Codes: A01M 7/0089 Regulating or controlling systems the delivery being related to the movement of a vehicle B05B9/06
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30188 Vegetation; Agriculture
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

An example machinery includes an automated crop management motorized vehicle having an intelligent, modularized image sensor (e.g. camera or video) system that is portable to other crop management vehicles such as a combine, planter or a tillage machine. The image sensor system includes a framework having a bank of procedures for monitoring and control of navigation, spray application, weeding, seeding, machine configuration, in real time as the machines go through a crop field throughout a crop cycle. One example implementation includes electronic circuits, with more than one set mounted on a platform that facilitates moving the setup to other agricultural machines. The framework captures, preserves and corrects the captured images for real time analysis and response, and for spray management to improve crop yield that is correlated with the machine settings and crop management practices.

Claims

1. An automated agricultural spray vehicle having an image sensor system comprising:

a modularized plug and play unit relocatable between a crop management machine and the automated agricultural spray vehicle;

wherein the modularized plug and play unit includes multiple image sensors mounted along a common platform;

wherein the common platform is portably mounted onto a boom on the autonomous agricultural spray vehicle;

each of the multiple image sensors having a non-transitory computer-readable circuit and memory encoded with rule-based instructions and artificial-intelligent instructions;

wherein each of the multiple image sensors is programmed to record images periodically at a rate proportional to a speed of the automated agricultural spray vehicle;

wherein recorded images are divided into a grid map, wherein each grid member depicts a representative particular field area;

wherein the modularized plug and play unit further includes a trigger system based on at least a color signal intensity surpassing a pre-determined threshold, projects an artificial-intelligent output chemical pattern; and

wherein at least one of the multiple image sensors is in electrical communication with a corresponding spray nozzle attached to the boom, and wherein the spray nozzle releases the chemical pattern responsive to a decision of the trigger system.

2. The automated agricultural spray vehicle of claim 1, wherein the spray nozzle releases a chemical mixture in the artificial-intelligent output chemical pattern using a trained U-Net neural network architecture.

3. An autonomous agricultural spray vehicle having an image sensor system comprising:

a modularized plug and play unit that relocates between a crop management machine and an automated sprayer machine;

the modularized plug and play unit including multiple image sensors mounted along a common platform;

the common platform portably mounted lengthwise parallel to a spray boom on a motor vehicle of the automated sprayer machine;

each of the multiple image sensors set to capture images periodically at a rate proportional to a speed of travel of the motor vehicle;

each of the multiple image sensors having a non-transitory computer-readable circuit and memory encoded with instructions and capturable images;

the capturable images divided into a grid map, and each grid member depicting a representative particular field area;

the modularized plug and play unit further including a trigger system that responds based on at least a peak value or one grid member having a color signal intensity surpassing a pre-determined threshold; and

wherein at least one of the multiple image sensors is in electrical communication with a corresponding spray nozzle attached to the spray boom, and wherein the spray nozzle releases a chemical responsive to a decision of the trigger system.

4. The image sensor system of claim 3, wherein the spray nozzle releases the chemical in a desired spray pattern based on a trained neural net having an U-Net architecture, and wherein the trained neural net maps representative field areas to the desired spray pattern.

5. The image sensor system of claim 3, wherein images from adjacent image sensors are stitched together, forming a redundant overlapping representation of a strip of a field area;

and wherein a redundant member is checked for a false trigger based on an image from another one of the adjacent image sensors.

6. The image sensor system of claim 5, wherein the trigger system responds based on any one of the redundant members surpassing the pre-determined threshold, regardless whether any of the adjacent image sensors has a sensor failure.

7. The image sensor system of claim 3, wherein the instructions stop searching for a further weed candidate among grid members after one candidate surpasses the pre-determined threshold in a spray area covered by one spray release.

8. The image sensor system of claim 3, further comprising a first program that analyzes colored grid members that align within three sigma to calculate a position of a crop row on the grid map;

and wherein the first program further analyzes a distance between colored grid members associated with the crop row and a grid member that is associated with a candidate weed.

9. The image sensor system of claim 3, wherein data collected from a previous pass of a seed planter machine through the field are stored in memory circuits of the modularized plug and play unit;

and wherein the trigger system is programmed to make a subsequent decision in a later pass through the field by the automated sprayer machine.

10. The image sensor system of claim 9, further comprising a second program that looks up stored data about a location of a planted seed on the grid map and associates a stored location datum with an associated seed grid member;

and wherein the trigger system is further based on the at least one grid member surpassing a pre-determined distance from the associated seed grid member.

11. The image sensor system of claim 3, wherein the representative particular field area is located ahead of the spray boom in a direction of travel of the motor vehicle;

and wherein each of the multiple image sensors also includes a rearward looking sensor that captures a rearward image, away from the direction of travel.

12. The image sensor system of claim 11, further comprising a third program that analyzes a captured rearward image using a color difference and a reflectance difference to distinguish between spray released from a spray nozzle and a drift of the released spray, wherein the color difference is based on an a priori color calibration of a background field without spray and of a lighting condition.

13. The image sensor system of claim 11, further comprising a fourth program that analyzes the captured rearward image using a color difference to detect spread patterns of dry fertilizer released from a spray nozzle, wherein the color difference is based on differences in color among grid members in the rearward image;

and wherein the spray nozzle is mounted on the spray boom.

14. The image sensor system of claim 11, further comprising a fifth program that analyzes the captured rearward image to detect a spray pattern skipping or patchiness among grid members representing a ground region behind the motor vehicle, based on a reflectance difference among the grid members.

15. The image sensor system of claim 3, wherein the multiple image sensors include two adjacent image sensors spaced apart on the common platform by a crop row spacing;

and the trigger system includes a fifth program that estimates a depth distance to form a 3-D map in a forward direction using images from the two adjacent image sensors.

16. The image sensor system of claim 15, wherein the trigger system analyzes the 3-D map to instruct the spray boom to adjust its position in response to terrain changes in an upcoming crop surface.

17. An autonomous agricultural spray vehicle having an image sensor system comprising:

a modularized plug and play unit relocating the modularized plug and play unit from a crop management machine to the autonomous agricultural spray vehicle;

wherein the modularized plug and play unit includes multiple image sensors mounted at a regular interval along a common platform;

wherein the common platform is portably mounted lengthwise parallel onto a boom on the autonomous agricultural spray vehicle;

each of the multiple image sensors having a non-transitory computer-readable circuit and memory encoded with instructions and data;

wherein each of the multiple image sensors is programmed to record images periodically at a rate proportional to a speed of the autonomous agricultural spray vehicle;

wherein recorded images are divided into a grid map, wherein each grid member depicts a representative particular ground area ahead of the autonomous agricultural spray vehicle;

wherein the modularized plug and play unit further includes a trigger system based on at least a peak color signal intensity surpassing a pre-determined threshold; and

wherein at least one of the multiple image sensors is in electrical communication with a corresponding spray nozzle attached to the boom, and wherein the spray nozzle releases a chemical responsive to a decision of the trigger system.

18. The autonomous agricultural spray vehicle of claim 17, wherein the representative particular ground area is located ahead of the boom in a direction of travel of the autonomous agricultural spray vehicle;

and wherein each of the multiple image sensors also includes a rearward looking sensor that records a rearward image, away from the direction of travel; and

the instructions include analyzing the recorded rearward image to detect a spray pattern skipping from a discontinuity among grid members representing a ground region behind the autonomous agricultural spray vehicle, based on a value of reflectance difference among grid members.

19. The autonomous agricultural spray vehicle of claim 17, wherein each of the multiple image sensors is in wireless communication with a central computer;

and wherein each of the multiple image sensors downloads a new agricultural algorithm relevant; and

wherein each of the multiple image sensors periodically uploads the recorded images.

20. The autonomous agricultural spray vehicle of claim 17, wherein at least one of the multiple image sensors includes a lens filter that sharpens a contrast between a desired color and other colors;

wherein a targeted object has the desired color.