Tagging of fruit-producing flowers for robotic selective harvesting

• US 9,532,508 B1
• Filed: 04/27/2015
• Issued: 01/03/2017
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Mineral Earth Sciences LLC
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: McCloskey, Kevin James Fleming, Stubbs, Mark Adrian
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US14/697,375
• Time in Patent Office: 617 Days
• US Class Current: 1/1
• CPC Class Codes: A01B 79/005 Precision agriculture
• CPC Class Codes: A01D 46/30 Robotic devices for individually picking crops
• CPC Class Codes: A01D 91/04 Products growing above the soil fruit, hops A01D46/00
• CPC Class Codes: A01G 22/05 Fruit crops, e.g. strawberries, tomatoes or cucumbers
• CPC Class Codes: A01G 9/006 Labels or label holders specially adapted for plant receptacles
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/0425 Safety, monitoring
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/31 Gripping jaw
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The locations of flowers on a plant, rather than the locations of agricultural products produced from such flowers, are used to facilitate the performance of harvesting and other agricultural operations in robotic agricultural applications. In some implementations, the identified location of a fruit-producing flower may be used by a robotic device to apply an indicator tag to a flowering plant proximate the flower for later identification when performing various types of directed and automated agricultural operations. In other implementations, the identified location of a fruit-producing flower may be used by a robotic device to anchor a stem of a flowering plant to a predetermined location such that the location of the flower, and of any fruit(s) later produced by such flower, are controlled and/or known when performing subsequent agricultural operations.

Claims

1. A method, comprising:

receiving digital image data of at least a portion of a flowering plant;

processing the received digital image data to identify a location of a fruit-producing flower on the flowering plant; and

prior to production of a fruit by the fruit-producing flower, controlling an end-effector of a robotic device to apply an indicator tag to a stem of the flowering plant proximate the identified location of the fruit-producing flower, the indicator tag being configured for detection by a machine perception system to identify a location of the fruit produced by the fruit-producing flower.

2. The method of claim 1, further comprising:

locating the indicator tag with the machine perception system; and

performing a directed and automated agricultural operation on the flowering plant proximate the indicator tag.

3. The method of claim 2, wherein performing the directed and automated agricultural operation includes harvesting the fruit.

4. The method of claim 3, wherein harvesting the fruit includes manipulating the indicator tag to sever the stem and thereby separate the fruit from the flowering plant.

5. The method of claim 4, wherein the indicator tag includes a collapsible loop that circumscribes the stem, and wherein manipulating the indicator tag includes collapsing the collapsible loop to sever the stem.

6. The method of claim 5, wherein the indicator tag includes a cable tie portion including a cable that projects through a catch to define the collapsible loop, wherein controlling the end-effector of the robotic device to apply the indicator tag includes drawing the cable through the catch to a first position, and wherein manipulating the indicator tag to sever the stem includes drawing the cable further through the catch beyond the first position.

7. The method of claim 6, wherein the cable tie portion includes at least one cutting element configured to cut the stem when the cable is further drawn through the catch.

8. The method of claim 2, wherein at least a portion of the indicator tag is reflective, and wherein locating the indicator tag includes detecting a reflection of electromagnetic radiation with the machine perception system.

9. The method of claim 8, wherein locating the indicator tag further includes emitting the electromagnetic radiation with the machine perception system.

10. The method of claim 8, wherein the indicator tag includes a reflector element that is reflective in multiple directions.

11. The method of claim 10, wherein the indicator tag is configured to hang the reflector element below the stem and below an occluding structure of the flowering plant.

12. The method of claim 2, wherein the directed and automated agricultural operation includes pollinating the fruit-producing flower, capping the fruit-producing flower to inhibit pollination, transfecting the fruit-producing flower, performing a targeted application of an agricultural fertilizer, pesticide or herbicide to the flowering plant, moving and affixing the stem of the flowering plant to a predetermined location, monitoring growth of the fruit, or determining ripeness of the fruit.

13. The method of claim 2, further comprising storing the location of the fruit-producing flower, wherein locating the indicator tag with the machine perception system includes retrieving the location of the fruit-producing flower to narrow a search space for locating the indicator tag with the machine perception system.

14. The method of claim 2, wherein the robotic device is a first robotic device, and wherein locating the indicator tag and performing the directed and automated agricultural operation on the flowering plant are performed by the first robotic device or a second robotic device.

15. The method of claim 1, wherein receiving the digital image data and processing the received digital image data are performed by the robotic device.

16. The method of claim 1, wherein the fruit comprises a vegetable agricultural product, a fruit agricultural product, or a nut agricultural product.

17. The method of claim 1, wherein the indicator tag comprises a Volatile Organic Compound (VOC) sensor.

18. An apparatus, comprising:

a robotic device including an end-effector configured to apply an indicator tag to a flowering plant; and

a control system coupled to the robotic device and configured to receive digital image data of at least a portion of the flowering plant, process the received digital image data to identify a location of a fruit-producing flower on the flowering plant, and prior to production of a fruit by the fruit-producing flower, control the end-effector to apply the indicator tag to a stem of the flowering plant proximate the identified location of the fruit-producing flower, wherein the indicator tag is configured for detection by a machine perception system to identify a location of a fruit produced by the fruit-producing flower on the flowering plant.

19. The apparatus of claim 18, wherein the robotic device further includes an image capture device configured to collect the digital image data.

20. A method, comprising:

receiving digital image data of at least a portion of a flowering plant;

processing the received digital image data to identify a location of a fruit-producing flower on the flowering plant;

prior to production of a fruit by the fruit-producing flower, anchoring a stem of the flowering plant proximate the identified location of the fruit-producing flower to a predetermined location using a robotic device; and

after production of the fruit by the fruit-producing flower, performing an agricultural operation on the flowering plant with the stem of the flowering plant anchored in the predetermined location.

21. The method of claim 20, wherein the agricultural operation includes harvesting the fruit with the stem of the flowering plant anchored in the predetermined location.

22. The method of claim 21, further comprising processing received digital image data for a plurality of flowering plants to identify locations of a plurality of fruit-producing flowers and using the identified locations to control an end-effector of the robotic device to anchor a plurality of stems proximate the plurality of fruit-producing flowers to a plurality of predetermined locations, wherein performing the agricultural operation includes performing a cutting operation along a predetermined path between the plurality of predetermined locations to harvest fruit from the plurality of flowering plants.