AUTOMATIC PAINTING SYSTEM WITH DRONE, USER INTERFACE AND COMPUTER VISION

• US 20170259920A1
• Filed: 03/10/2016
• Published: 09/14/2017
• Est. Priority Date: 03/10/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lai, Jui-Hsin, Ma, Yu, Shi, Conglei, Xia, Yinglong
• Granted Patent: US 10,023,311 B2
• Time in Patent Office: Days
• CPC Class Codes: B64D 1/18 by spraying, e.g. insecticides spraying apparatus per se B05B
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/28 for painting or marking
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A system and method for painting a structure. The system comprises: a computer vision processing system configured to obtain images of a target structure and generate first instructions signals for real time communication to an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon; a control device at the UAV responsive to received the first instruction signals for controlling real time navigating of the UAV to a location at the target structure; and the computer vision processing system configured to generate second instruction signals for real time communication to the UAV, wherein the control device at the UAV is configured to automatically actuate the paint fluid dispensing system to apply a paint fluid at the location on the target structure in response, the received second instruction signals configuring the UAV to render a desired visual image on the target structure.

Claims

1. A painting system comprising:

an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon;

a controller device at said UAV configured to receive instruction signals for controlling real time navigating of said UAV to a target location at said target structure; and

said UAV further responsive to the controller device to configure said UAV to navigate said target structure by traversing said structure along a flight path, and

said UAV further responsive to the controller device to configure said UAV to automatically actuate said paint fluid dispensing system to apply paint fluid at a location on the target structure, wherein said UAV paints a desired visual image on said target structure.

2. The painting system as claimed in claim 1, further comprising:

a mobile device base station running a computer vision processing system configured to obtain real-time images of said UAV relative to the target location and for generating said instructions signals for real time navigating of said UAV and communicating said instructions signals to said UAV.

3. The painting system as claimed in claim 1, wherein said computer vision processing system comprises an image capture device for initially obtaining an image of the target structure,

said mobile device comprising:

a processor device configured to run an application to edit said obtained image of the target structure and overlay the desired visual image onto said obtained image via a display of said mobile device base station, wherein said target location corresponds to a location where said desired visual image is to be painted.

4. The painting system as claimed in claim 1, wherein said processor device at said mobile device is further configured to:

run an application to generate flight path information for said UAV to traverse said target structure according to said flight path, said generated flight path information including initial location coordinates and final location coordinates corresponding to a pixel coordinate system of said mobile device base station display.

5. The painting system as claimed in claim 3, wherein said image capture device is configured to:

capture an image of the target structure having said painted desired visual image thereon, said processor device at said mobile phone further configured to;

compare the captured image of the target structure having the painted visual image thereon with the initially obtained image having the corresponding desired visual image overlayed thereon to determine need for additional painting by said UAV on the target structure.

6. The painting system as claimed in claim 5, wherein responsive to determining a need for additional painting by said UAV, said processor device of said mobile phone device configured to:

communicate wireless signals for receipt at said UAV to navigate said UAV to a particular location at the target structure, and automatically activate said UAV to traverse said target structure and activate said paint fluid dispensing system to apply a paint color to said particular location at said target structure according to said desired visual image.

7. The painting system as claimed in claim 6, wherein based on comparison results between the visual image painted by the UAV on said target structure with a corresponding desired visual image overlay of the obtained image at said mobile device display, automatically determine one or more difference regions on the target structure, wherein the UAV is navigated along a path to traverse a difference region to modify said image within said difference region.

8. The painting system as claimed in claim 6, wherein said mobile device base station and said UAV is configured to iteratively repeat said navigating to a target position, said traversing said target structure along the flight path, and said activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure until the desired visual image is painted on said target structure.

9. The painting system as claimed in claim 4, wherein to generate said flight path information, said controller device at said UAV is configured to:

receive a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display;

receive a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determine, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.

10. A method for painting a target structure comprising:

receiving, at a processor device of an unmanned aerial vehicle (UAV), navigation instruction signals for real time navigating said UAV to a target location at the target structure, said UAV having a paint fluid dispensing system provided thereon;

using instructions signals from said processor device to configure said UAV to traverse said target structure along a flight path; and

using further instructions signals from said processor device to configure said UAV to automatically activate said paint fluid dispensing system during said traversal along the flight path to apply a paint fluid on the target structure, wherein said UAV paints a desired visual image on said target structure.

11. The method as claimed in claim 10, wherein prior to said real time navigating said UAV:

obtaining, via a computer vision processing system running at a mobile device base station, real-time images of said UAV relative to the target structure;

generating said navigation instructions signals UAV and communicating said instructions signals to said UAV for real time navigating said UAV to the target location.

12. The method as claimed in claim 10, wherein said computer vision processing system at said mobile base station comprises an image capture device for initially obtaining an image of the target structure, said method comprising:

obtaining, via the image capture device, an image of the target structure to be painted; and

overlaying, using an image editing program, a desired visual image on said obtained image of said target structure via a display interface of said mobile device base station.

13. The method as claimed in claim 10, further comprising:

generating, via a processor device at said mobile device, flight path information for said UAV, said flight path information configuring said UAV to traverse said target structure according to said flight path, said generated flight path information including initial position coordinates and final position coordinates corresponding to a pixel coordinate system of said mobile device base station display interface.

14. The method as claimed in claim 12, wherein the processor device of said mobile device performs:

capturing, via said image capture device, an image of the target structure having said painted desired visual image thereon;

comparing the captured image of the target structure having the visual image painted thereon by the UAV thereon with the obtained image having the corresponding desired visual image overlay to determine need for additional painting by said UAV on the target structure.

15. The method as claimed in claim 13, further comprising:

responsive to determining a need for additional painting by said UAV on the target structure;

communicating wireless signals from said mobile device base station for receipt at said UAV to navigate said UAV to a particular location at the target structure; and

automatically activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure corresponding to said desired visual image.

16. The method as claimed in claim 13, wherein the generating of flight path information for said UAV comprises:

receiving, at a controller of said UAV, a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display interface;

receiving a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determining, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.

17. The method as claimed in claim 15, wherein based on comparison results between the visual image painted by the UAV on said target structure with a corresponding desired visual image overlay of the obtained image at said mobile device display,

automatically determining one or more difference regions on the target structure, wherein the UAV is navigated along a path to traverse a difference region to modify said image within said difference region.

18. The method as claimed in claim 15, further comprising:

iteratively repeating said navigating said UAV to a target position, said traversing said target structure along the flight path, and said activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure until the desired visual image is painted on said target structure.

19. A method comprising:

configuring a computer vision processing system running at a mobile device base station to obtain real-time images of an unmanned aerial vehicle (UAV) relative to a target structure;

generating, by a processor device running at said mobile device base station, navigation instructions signals for real time navigating said UAV to a target location at the target structure, said UAV having a paint fluid dispensing system provided thereon;

generating, by the processor device running at said mobile device base station, a flight path for said UAV to traverse said target structure and communicating said instructions signals to navigate said UAV to traverse said target structure along the flight path; and

wherein said further instructions signals from said processor device are used to configure said UAV to automatically activate said paint fluid dispensing system during said traversal along the flight path to apply a paint fluid on the target structure, wherein said UAV paints a desired visual image on said target structure.

20. The method as claimed in claim 19, wherein said computer vision processing system comprises an image capture device, wherein prior to said real time navigating said UAV, said method further comprising:

initially obtaining an image of the target structure via said image capture device;

editing, at said mobile device base station, said obtained image of the target structure to overlay the desired visual image onto said obtained image via a display of said mobile device base station, wherein said target location corresponds to a location where said desired visual image is to be painted.

21. The method as claimed in claim 19, wherein the generating of a flight path for said UAV comprises:

communicating, from the mobile device base station to a controller device of said UAV, a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display interface;

communicating, from the mobile device base station to a controller device of said UAV, a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;

determining, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;

compare the pixel distance to a predetermined pixel distance threshold; and

program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.