Automatic painting system with drone, user interface and computer vision

• US 10,023,311 B2
• Filed: 03/10/2016
• Issued: 07/17/2018
• Est. Priority Date: 03/10/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lai, Jui-Hsin, Ma, Yu, Shi, Conglei, Xia, Yinglong
• Primary Examiner(s): Mawari, Redhwan K
• Assistant Examiner(s): King, Rodney P
• Application Number:
  US15/066,708
• Publication Number:
  US 20170259920A1
• Time in Patent Office: 859 Days
• Field of Search: 701 2
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 1/18 by spraying, e.g. insecticides spraying apparatus per se B05B
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/28 for painting or marking
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A system and method for painting a structure. The system comprises: a computer vision processing system configured to obtain images of a target structure and generate first instructions signals for real time communication to an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon; a control device at the UAV responsive to received the first instruction signals for controlling real time navigating of the UAV to a location at the target structure; and the computer vision processing system configured to generate second instruction signals for real time communication to the UAV, wherein the control device at the UAV is configured to automatically actuate the paint fluid dispensing system to apply a paint fluid at the location on the target structure in response, the received second instruction signals configuring the UAV to render a desired visual image on the target structure.

Claims

1. A painting system comprising:

a mobile device having a processor configured to receive a digital image of a target structure, said received digital image of said target structure including an overlay of a visual image desired to be physically rendered on said target structure;

an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon;

a controller device at said UAV configured to receive instruction signals for controlling real time navigating of said UAV to a target location at said target structure;

said processor of said mobile device obtaining real-time images of said UAV relative to the target structure for recognizing a location of said UAV relative to said target structure and for generating said instruction signals

to configure said UAV to navigate said target structure by traversing said structure along a flight path, and

said processor of said mobile device generating further instruction signals

to configure said UAV controller to automatically actuate said paint fluid dispensing system to apply paint fluid at a location on the target structure corresponding to said visual image overlay of said received digital image while navigating said target structure along said flight path, said processor of said mobile device using said obtained real-time images of said UAV relative to the target location for real-time refining said instruction signals for traversing said target structure and refining said further instruction signals for controlling said paint fluid dispensing system to apply said paint, wherein said UAV paints the desired visual image on said target structure.

2. The painting system as claimed in claim 1, wherein said processor of said mobile device runs a computer vision processing system configured to obtain said real-time images of said UAV relative to the target location and for said generating and refining said instructions signals for real time navigating of said UAV and communicating said instructions signals to said UAV and for said generating and refining said further instruction signals for controlling said paint fluid dispensing system to apply said paint and communicating said further instruction signals to said UAV.

3. The painting system as claimed in claim 2, wherein said computer vision processing system comprises an image capture device for initially obtaining said digital image of the target structure,

said mobile device comprising:

a processor device configured to run an application to edit said obtained digital image of the target structure and receive, from a user, said overlay of the desired visual image onto said obtained digital image via a display of said mobile device.

4. The painting system as claimed in claim 3, wherein said processor device at said mobile device is further configured to:

run an application to generate flight path information for said UAV to traverse said target structure according to said flight path, said generated flight path information including initial location coordinates and final location coordinates corresponding to a pixel coordinate system of said mobile device display.

5. The painting system as claimed in claim 4, wherein to generate said flight path information, said controller device at said UAV is configured to:

receive a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display;

receive a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determine, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.

6. The painting system as claimed in claim 3, wherein said image capture device is configured to:

capture an image of the target structure having said painted desired visual image thereon, said computer vision processing at said mobile device further configured to;

compare the captured image of the target structure having the painted visual image thereon with the received digital image of the target structure having the corresponding desired visual image overlayed thereon to determine need for additional painting by said UAV on the target structure.

7. The painting system as claimed in claim 6, wherein responsive to determining a need for additional painting by said UAV, said processor device of said mobile device configured to:

communicate wireless instruction signals for receipt at said UAV to navigate said UAV to a particular location at the target structure, and automatically activate said UAV to traverse said target structure and activate said paint fluid dispensing system to apply a paint color to said particular location at said target structure according to said desired visual image.

8. The painting system as claimed in claim 7, wherein based on comparison results between the visual image painted by the UAV on said target structure with a corresponding desired visual image overlay of the received digital image at said mobile device display, automatically determine one or more difference regions on the target structure, wherein the UAV is navigated along a path to traverse a difference region to modify said image within said difference region.

9. The painting system as claimed in claim 7, wherein said mobile device and said UAV is configured to iteratively repeat said navigating to a target position, said traversing said target structure along the flight path, and said activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure until the desired visual image is painted on said target structure.

10. A method for painting a target structure comprising:

receiving, at a processor of a mobile device, a digital image of a target structure, said received digital image of said target structure including an overlay of a visual image desired to be physically rendered on said target structure;

configuring a controller device of an unmanned aerial vehicle (UAV) to receive navigation instruction signals for real time navigating said UAV to a target location at the target structure, said UAV having a paint fluid dispensing system provided thereon;

obtaining, at said processor of said mobile device, real-time images of said UAV relative to the target structure for recognizing a location of said UAV relative to said target structure and for generating said navigation instructions

to configure said UAV to traverse said target structure along a flight path; and

generating, by said processor of said mobile device, further instruction signals

to configure said UAV controller device to automatically activate said paint fluid dispensing system during said traversal along the flight path to apply a paint fluid on the target structure corresponding to said visual image overlay of said received digital image while navigating said target structure along said flight path, said processor of said mobile device using said obtained real-time images of said UAV relative to the target location for real-time refining said navigation instruction signals for traversing said target structure and refining of said further instruction signals for controlling said paint fluid dispensing system to apply said paint, wherein said UAV paints the desired visual image on said target structure.

11. The method as claimed in claim 10, further comprising:

configuring, by said processor of said mobile device, a computer vision processing system to obtain said real-time images of said UAV relative to the target structure and for said

generating and refining said navigation instructions signals for said real-time navigating of said UAV and communicating said instructions signals to said UAV for real time navigating said UAV to the target location and for said generating and refining said further instruction signals for controlling said paint fluid dispensing system to apply said paint and communicating said further instruction signals to said UAV.

12. The method as claimed in claim 11, wherein said computer vision processing system at said mobile device comprises an image capture device for initially obtaining said digital image of the target structure, said method comprising:

obtaining, via the image capture device, an image of the target structure to be painted; and

overlaying by a user, using an image editing program, said desired visual image on said obtained digital image of said target structure via a display interface of said mobile device.

13. The method as claimed in claim 12, further comprising:

generating, via the processor device at said mobile device, flight path information for said UAV, said flight path information configuring said UAV to traverse said target structure according to said flight path, said generated flight path information including initial position coordinates and final position coordinates corresponding to a pixel coordinate system of said mobile device display interface.

14. The method as claimed in claim 13, wherein the generating of flight path information for said UAV comprises:

receiving, at the controller device of said UAV, a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display interface;

receiving a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determining, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.

15. The method as claimed in claim 12, wherein the processor device of said mobile device performs:

capturing, via said image capture device, an image of the target structure having said painted desired visual image thereon; and

comparing, using said computer vision processing system, the captured image of the target structure having the visual image painted thereon by the UAV thereon with the received digital image having the corresponding desired visual image overlay to determine need for additional painting by said UAV on the target structure.

16. The method as claimed in claim 15, further comprising:

responsive to determining a need for additional painting by said UAV on the target structure;

communicating wireless instruction signals from said mobile device for receipt at said UAV to navigate said UAV to a particular location at the target structure; and

automatically activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure corresponding to said desired visual image.

17. The method as claimed in claim 16, wherein based on comparison results between the visual image painted by the UAV on said target structure with a corresponding desired visual image overlay of the received digital image at said mobile device display,

automatically determining one or more difference regions on the target structure, wherein the UAV is navigated along a path to traverse a difference region to modify said image within said difference region.

18. The method as claimed in claim 16, further comprising:

iteratively repeating said navigating said UAV to a target position, said traversing said target structure along the flight path, and said activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure until the desired visual image is painted on said target structure.

19. A method comprising:

initially obtaining, via an image capture device of a mobile device, a digital image of a target structure, said obtained digital image of the target structure including an overlay of a desired visual image via a display of said mobile device;

configuring a computer vision processing system running at a mobile device to obtain real-time images of an unmanned aerial vehicle (UAV) relative to a target structure;

generating, by a processor device running at said mobile device base station, navigation instructions signals for real time navigating said UAV to a target location at the target structure, said UAV having a paint fluid dispensing system provided thereon, said target location corresponding to a location where said desired visual image is to be painted;

generating, by the processor device running at said mobile device base station, a flight path for said UAV to traverse said target structure and communicating said instructions signals to navigate said UAV to traverse said target structure along the flight path; and

generating further instructions signals from said processor device to configure said UAV to automatically activate said paint fluid dispensing system during said traversal along the flight path to apply a paint fluid on the target structure, wherein said UAV paints said desired visual image on said target structure.

20. The method as claimed in claim 19, wherein prior to said real time navigating said UAV, said method further comprising:

editing by a user, at said mobile device, said obtained digital image of the target structure to overlay the desired visual image onto said obtained digital image of said target structure via a display of said mobile device.

21. The method as claimed in claim 19, wherein the generating of a flight path for said UAV comprises:

communicating, from the mobile device base station to a controller device of said UAV, a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display interface;

communicating, from the mobile device base station to a controller device of said UAV, a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determining, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.