Apparatuses and methods for accurate structure marking and marking-assisted structure locating

• US 10,328,411 B2
• Filed: 09/25/2017
• Issued: 06/25/2019
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Singh, Navrit Pal
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US15/714,309
• Publication Number:
  US 20180021751A1
• Time in Patent Office: 638 Days
• CPC Class Codes: B01J 19/121 Coherent waves, e.g. laser beams lasers per se H01S3/00
• CPC Class Codes: B23K 26/0626 Energy control of the laser beam B23K26/0622 takes precedence
• CPC Class Codes: B23K 26/082 Scanning systems, i.e. devices involving movement of the laser beam relative to the laser head
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G05B 19/40937 concerning programming of machining or material parameters, pocket machining
• CPC Class Codes: G05B 2219/32025 Automatic marking of article
• CPC Class Codes: G05B 2219/45134 Marking
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

Working equipment includes a tool configured to work a structure at a working location thereon, with the structure having an applied marking at a known location with a known relationship with the working location. A computer system is configured to determine placement of the structure, and accordingly position the tool into at least partial alignment with the working location, and which in at least one instance, the tool is aligned with a second, offset location. A camera is configured to capture an image of the structure and including the marking, and further including the second location with which the tool is aligned. And the computer system is configured to process the image to locate the working location, reposition the tool from the second location and into greater alignment with the located working location, and control the repositioned tool to work the structure at the located working location.

Claims

1. A working equipment comprising:

a tool configured to work a structure at a working location thereon, the structure having a permanent marking applied thereto at a known location with a known relationship with the working location;

a computer system coupled to the tool and configured to determine placement of the structure, and position the tool relative to the working location according to the placement, the tool in at least one instance being aligned with a second location offset from the working location; and

a camera coupled with the computer system and configured to capture an image of at least a portion of the structure and including the permanent marking, and further including the second location with which the tool is aligned,

wherein the computer system is configured to process the image to locate the working location, reposition the tool from the second location to a location more closely aligned with the located working location than the second location, and control the repositioned tool to work the structure at the located working location.

2. The working equipment of claim 1, wherein the tool being configured to work the structure includes being configured to work the structure in a manner of a plurality of manners including drill a hole, install a fastener, cut the structure or route out a portion of the structure.

3. The working equipment of claim 2, wherein a geometry of the permanent marking indicates the manner in which the tool is to work the structure, and

wherein the computer system being configured to process the image includes being configured to process the image to also determine the manner in which the tool is to work the structure, the computer system being configured to control the repositioned tool to work the structure according to the manner so determined.

4. The working equipment of claim 1, wherein the camera is a second camera configured to capture a second image, and the working equipment further comprises:

a first camera coupled to the computer system and configured to capture a first image of at least a portion of the structure and including one or more targets on or proximate the structure,

wherein the computer system is configured to process the first image to thereby determine the placement.

5. The working equipment of claim 4, wherein the computer system being configured to process the first image includes being configured to process the first image using a design master file for the structure, the design master file including edge-of-structure information that relates to structure geometry that defines the structure, calibration point information that provides known locations of the one or more targets, and location information that identifies the working location.

6. The working equipment of claim 1 further comprising:

a laser scanner configured to scan the structure and provide measurements of points on a surface of the structure,

wherein the computer system is configured to process the measurements to generate a three-dimensional (3D) model of the structure, and process the 3D model to thereby determine the placement.

7. The working equipment of claim 1, wherein the computer system being configured to reposition the tool includes being configured to reposition the tool only when the offset is greater than a predetermined threshold.

8. The working equipment of claim 1 further comprising:

a movable end effector assembly coupled to the computer system, and including an end effector and the tool, wherein the camera is secured to the end effector assembly, and

wherein the computer system being configured to position the tool includes being configured to position the end effector assembly and thereby the tool, the camera also being thereby positioned such that a field of view of the camera encompasses the permanent marking.

9. The working equipment of claim 1, wherein the camera has a field of view divided into a plurality of concentric zones of increasing size about the second location with which the tool is aligned, the zones including a first zone that defines an acceptable offset, and a larger second zone located outside the first zone that defines an unacceptable offset, and

wherein the computer system is configured to control the tool to work the structure without repositioning in an instance in which the located working location is within the first zone, or reposition the tool before controlling the tool to work the structure in an instance in which the located working location is within the second zone.

10. The working equipment of claim 9, wherein the zones further include a third zone that is larger than and located outside the first zone and the second zone, and

wherein the computer system is configured to produce an error notification without the tool being repositioned or controlled to work the structure in an instance in which the located working location is within the third zone.

11. A method comprising:

determining placement of a structure having a permanent marking applied thereto at a known location with a known relationship with a working location thereon;

positioning a tool relative to the working location according to the placement, the tool in at least one instance being aligned with a second location offset from the working location;

capturing an image of at least a portion of the structure and including the permanent marking, and further including the second location with which the tool is aligned; and

processing the image to locate the working location, repositioning the tool from the second location to a location more closely aligned with the located working location than the second location, and controlling the repositioned tool to work the structure at the located working location.

12. The method of claim 11, wherein controlling the repositioned tool to work the structure includes controlling the repositioned tool to work the structure in a manner of a plurality of manners including drill a hole, install a fastener, cut the structure or route out a portion of the structure.

13. The method of claim 12, wherein a geometry of the permanent marking indicates the manner in which the tool is to work the structure, and

wherein processing the image includes processing the image to also determine the manner in which the tool is to work the structure, the repositioned tool being controlled to work the structure according to the manner so determined.

14. The method of claim 11, wherein the image is a second image captured by a second camera, and wherein determining the placement comprises:

capturing by a first camera, a first image of at least a portion of the structure and including one or more targets on or proximate the structure; and

processing the first image to thereby determine the placement.

15. The method of claim 14, wherein processing the first image includes processing the first image using a design master file for the structure, the design master file including edge-of-structure information that relates to structure geometry that defines the structure, calibration point information that provides known locations of the one or more targets, and location information that identifies the working location.

16. The method of claim 11 further comprising:

scanning by a laser scanner, the structure and providing measurements of points on a surface of the structure;

processing the measurements to generate a three-dimensional (3D) model of the structure; and

processing the 3D model to thereby determine the placement.

17. The method of claim 11, wherein repositioning the tool includes repositioning the tool only when the offset is greater than a predetermined threshold.

18. The method of claim 11, wherein a movable end effector assembly includes an end effector and the tool, wherein the image is captured by a camera secured to the end effector assembly, and

wherein positioning the tool includes positioning the end effector assembly and thereby the tool, the camera also being thereby positioned such that a field of view of the camera encompasses the permanent marking.

19. The method of claim 11, wherein the image is captured by a camera having a field of view divided into a plurality of concentric zones of increasing size about the second location with which the tool is aligned, the zones including a first zone that defines an acceptable offset, and a larger second zone located outside the first zone that defines an unacceptable offset, and

wherein repositioning the tool and controlling the repositioned tool to work the structure includes controlling the tool to work the structure without repositioning in an instance in which the located working location is within the first zone, or repositioning the tool before controlling the tool to work the structure in an instance in which the located working location is within the second zone.

20. The method of claim 19, wherein the zones further include a third zone that is larger than and located outside the first zone and the second zone, and

wherein the method further comprises producing an error notification without the tool being repositioned or controlled to work the structure in an instance in which the located working location is within the third zone.