Augmented-human field inspection tools for automated phenotyping systems and agronomy tools

• US 10,638,667 B2
• Filed: 12/26/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/26/2017
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Regan, William, Bitterman, Matthew, Brown, David, Grant, Elliott, Yuan, Zhiqiang
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US15/854,607
• Publication Number:
  US 20190191630A1
• Time in Patent Office: 861 Days
• CPC Class Codes: A01B 79/005 Precision agriculture
• CPC Class Codes: A01G 22/00 Cultivation of specific crops or plants not otherwise provided for
• CPC Class Codes: A01G 7/00 Botany in general
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30188 Vegetation; Agriculture
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 19/003 Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 23/38 for botany
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

Systems and Methods for Augmented-Human Field Inspection Tools for Automated Phenotyping Systems and Agronomy Tools. In one embodiment, a method for plant phenotyping, includes: acquiring a first set of observations about plants in a field by a trainer. The trainer carries a sensor configured to collect observations about the plant, and the first set of observations includes ground truth data. The method also includes processing the first set of observations about the plants by a trait extraction model to generate instructions for a trainee; and acquiring a second set of observations about the plants by a trainee while the trainee follows the instructions.

Claims

1. A method for plant phenotyping, comprising:

acquiring a first set of observations about plants in a field by a trainer, wherein the trainer carries a sensor configured to collect observations about the plants, and wherein the first set of observations includes ground truth data;

recording a body position of the trainer while the trainer is collecting at least one of the first set of observations in the field;

recording a location of the trainer simultaneously with recording the body position;

processing the first set of observations about the plants by a trait extraction model to generate instructions for a trainee; and

acquiring a second set of observations about the plants by the trainee while the trainee follows the instructions.

2. The method of claim 1, wherein the trainer carries at least one camera configured to acquire images of the plants.

3. The method of claim 2, wherein the sensor is selected from a group consisting of a chemical sensor, a humidity sensor, an acoustic sensor, a distance sensor, and a GPS.

4. The method of claim 2, wherein the trainer carries a voice recorder, and wherein the first set of observations includes voice comments made in the field by the trainer, the method further comprising:

recording the voice comments by the voice recording equipment.

5. The method of claim 4, wherein the sensor, the camera, and the voice recorder operate simultaneously.

6. The method of claim 1, wherein the body position is at least one of bending toward the plants, reaching for the plants, or kneeling by the plants.

7. The method of claim 1, wherein the location of the trainer is determined at least in part based on a GPS signal.

8. The method of claim 2, wherein processing the trait extraction model comprises convolutional neural nets (CNN) configured to, when executed, analyze images.

9. The method of claim 4, wherein processing the trait extraction model comprises natural language processing (NLP) algorithms configured to, when executed, extract information from the voice comments by the first observer.

10. The method of claim 1, further comprising:

acquiring a second set of observations about the plants by the trainee, wherein the trainee carries an augmented reality (AR) system or a virtual reality (VR) system configured to at least partially display instructions for the trainee; and

processing the second set of observations about the plants by the trait extraction model to update the instructions.

11. The method of claim 1, further comprising:

based on the instructions, executing at least one plant treatment by the trainee.

12. The method of claim 1, further comprising:

based on the instructions, emulating the body position of the trainer by the trainee.

13. The method of claim 1, wherein the trainee is an automated observer system carried by an unmanned ground vehicle (UGV).

14. A system for plant phenotyping, comprising:

a handheld agronomist tool including;

a computing device having a memory that carries instructions for collecting observations about plants in a field;

at least one camera for observing the plants;

a first sensor coupled to the computing device and configured to record a body position of a trainer while the trainer collects at least one of a first set of observations in the field; and

a second sensor coupled to the computing device and configured to record a location of the trainer simultaneously with recording the body position;

wherein the instructions are generated by a trait extraction model based on processing the first set of observations about the plants collected by a trainer, wherein the instructions instruct a trainee to observe the plants or to execute plant treatments.

15. The system of claim 14, wherein the handheld agronomist further comprises a third sensor for collecting observations of the plants, and wherein the third sensor is one of a chemical sensor, a humidity sensor, an acoustic sensor, and a distance sensor.

16. The system of claim 15, wherein the trainee carries an augmented reality (AR) system or a virtual reality (VR) system configured to at least partially display instructions for the trainee.

17. The system of claim 15, wherein the trainee is an unmanned aerial vehicle (UAV), and wherein the handheld agronomist is carried by the UAV.

18. The system of claim 15, wherein the trainee is an unmanned ground vehicle (UGV), and wherein the handheld agronomist is carried by the UGV.

19. The system of claim 18, wherein the camera is a first camera, the system further comprising:

a second camera configured to operate in a range of wavelengths of electromagnetic (EM) radiation; and

an EM modulator that emits EM waves in the range of wavelengths.

20. The system of claim 18, further comprising an analysis system configured to execute the trait extraction model.