APPARATUS AND METHOD FOR FAULT-PROOF COLLECTION OF IMAGERY FOR UNDERWATER SURVEY
1. An apparatus for image collection, comprising:
- an underwater vehicle, comprisinga central motor comprising a rotating member;
a set of arms coupled to the rotating member and extending away from the central motor; and
one or more cameras attached to a distal end of at least one arm of the set of arms, wherein the underwater vehicle is configured to travel over a predetermined region and collect a set of images of the predetermined region.
An apparatus and method are presented comprising one or more sensors or cameras configured to rotate about a central motor. In some examples, the motor is configured to travel at a constant linear speed while the one or more cameras face downward and collect a set of images in a predetermined region of interest. The apparatus and method are configured for image acquisition with non-sequential image overlap. The apparatus and method are configured to eliminate gaps in image detection for fault-proof collection of imagery for an underwater survey. In some examples, long baseline (LBL) is utilized for mapping detected images to a location. In some examples, ultra-short baseline (USBL) is utilized for mapping detected images to a location. The apparatus and method are configured to utilize a simultaneous localization and mapping (SLAM) approach.
- 1. An apparatus for image collection, comprising:
an underwater vehicle, comprising a central motor comprising a rotating member; a set of arms coupled to the rotating member and extending away from the central motor; and one or more cameras attached to a distal end of at least one arm of the set of arms, wherein the underwater vehicle is configured to travel over a predetermined region and collect a set of images of the predetermined region.
- View Dependent Claims (2, 3, 4, 5, 6)
- 7. An apparatus for collection of images, comprising:
an underwater vehicle comprising a motor; and a set of sensors configured to rotate about the motor, wherein the underwater vehicle is configured to travel over a linear path in a predetermined region and detect information in the predetermined region.
- View Dependent Claims (8, 9, 10, 11, 12)
- 13. A method, comprising:
driving an underwater vehicle at a constant linear speed; rotating one or more cameras about a central motor; capturing a set of images of a region to be mapped; and detecting position of the underwater vehicle during image capture.
- View Dependent Claims (14, 15, 16, 17, 18, 19, 20, 21)
This application claims benefit under 35 U.S.C. § 119(e) to U.S. Provisional Application Ser. No. 62/752,070, titled “APPARATUS AND METHOD FOR FAULT-PROOF COLLECTION OF IMAGERY FOR UNDERWATER SURVEY,” filed on Oct. 29, 2018.
This invention was made with government support under National Oceanic and Atmospheric Administration (NOAA) grant number NA15NOS4000200. The government has certain rights in the invention.
The present invention relates to an apparatus and method for conducting underwater surveys, collection and processing of image and/or video data.
Underwater image and video surveys have become a common tool for researchers and commercial companies involved in cable and pipeline laying and inspection, building foundations for windfarms, and the like.
Researchers and experts have struggled to solve the SLAM problem in practical settings, at least in part because SLAM processes require a great deal of computational power to sense a sizable area and process the resulting data to both map and localize. A complete three-dimensional SLAM solution may be highly computationally intensive, for example, requiring complex real-time particle filters, sub-mapping strategies and/or hierarchical combination of metric topological representations.
Examples of the present disclosure meet one or more of the above needs, among others, by providing a system and method for collecting images and/or video, collecting position information associated with the images and/or video, and preparing of a map of an unknown environment using a computer to analyze the collected information. The map may be a topological map. The map may capture the environment by combining images of pieces of the environment and connecting them in an appropriate way. The map may provide details with improved geometric accuracy when compared to conventional systems and methods. The system and method disclosed herein may provide guaranteed high overlap between frames. The system and method disclosed herein may provide fault-proof collection of imagery. The system and method disclosed herein may be configured to provide a 2-dimensional (2D) or 2.5-dimensional (2.5D) representation of a predetermined region of interest.
Accordingly, pursuant to one aspect, there is provided an apparatus for image collection, comprising an underwater vehicle, comprising a central motor comprising a rotating member, a set of arms couples to the rotating member and extending away from the central motor, and one or more cameras attached to a distal end of at least one arm of the set of arms, wherein the underwater vehicle is configured to travel over a predetermined region and collect a set of images of the predetermined region.
Examples described herein may be further characterized by one or any combination of features, such as a processor is configured to create a full mosaic image using non-sequential image overlap. In some examples, at least one of the central motor, a set of arms, and one or more cameras provided with rounded edges to facilitate hydrodynamic motion. In some examples, the set of arms is housed within a disc. In some examples, the underwater vehicle is configured to utilize a long baseline (LBL) approach to determine positioning for each imaged location in the predetermined region. In some examples, the underwater vehicle is configured to utilize an ultra-short baseline (USBL) approach to determine positioning for each imaged location in the predetermined region.
Pursuant to another aspect, an apparatus for collection of images is provided. The apparatus comprises an underwater vehicle comprising a motor; and a set of sensors configured to rotate about the motor. The underwater vehicle may be configured to travel over a linear path in a predetermined region and detect information in the predetermined region.
Examples described herein may be further characterized by one or any combination of features, such as the set of sensors being a set of time-of-flight (TOF) sensors. In some examples, bathymetry information is processed. In some examples, the set of sensors includes a set of cameras. In some examples, the set of cameras are rotated at a constant rotational speed. In some examples, at least one of the set of sensors and the motor are housed within a disc.
Pursuant to yet another aspect, a method is provided. The method includes acts of driving an underwater vehicle at a constant linear speed, rotating one or more cameras about a central motor, capturing a set of images of a region to be mapped, and detecting position of the underwater vehicle during image capture.
Examples described herein may be further characterized by one or any combination of features, such as the one or more cameras are coupled to a processor configured for image co-registration using non-sequential image overlap. In some examples, one or more cameras that are rotated at a constant rotational speed, utilization of a long baseline (LBL) approach to detect position of the underwater vehicle in the region to be mapped, and utilization of an ultra-short baseline (USBL) approach to detect position of the underwater vehicle in the region to be mapped. In some examples, the one or more cameras are configured to send the set of images to a computer, to perform a bundle adjustment step, to perform global optimization utilizing SLAM and non-sequential image overlap, and/or to compile the set of images into a map of the region.
Further aspects, advantages and areas of applicability will become apparent from the description provided herein. The description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.
The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
The following description provides one or more examples but is not intended to limit the present disclosure, application, or uses. The present disclosure describes an apparatus that is configured to avoid typical errors for surveys that follow standard protocols. Typical surveys may include image surveys, video surveys, and/or acoustic surveys.
Advantages of a video survey include fast coverage of large areas, high spatial resolution of the data (acoustic surveys, for example, yield much lower resolution), ease of interpretation, and non-invasiveness, as compared to sample collection. Sample collection, as used herein, may refer to acquisition of a single sample grab of sediment or several sample grabs. Although sample collection is considered to provide a ground truth, this method has a number of deficiencies. First, although a sampler is dropped from a known position using GPS, the sampler is not guaranteed to hit the seafloor directly vertically below the known position due to currents, for instance. As a result of the imprecise positioning, the sampler may not provide adequate information. For example, the sampler may hit the only boulder in the otherwise sandy area. Second, areas of bedrock cannot be sampled at all. Third, sample collection is an invasive technique that may damage the environment. All of these issues are solved by utilizing the fault-proof collection of imagery for underwater survey as described herein.
Video surveys may typically be performed by Remotely Operated Vehicles (ROV), Autonomous Underwater Vehicles (AUV), from towed platforms, or just by divers with handheld cameras. The expected output is a detailed map, much larger than a single camera footprint, often with height information (2.5D), and preferably tied to a geographic location.
As will be seen, the devices and methods taught herein offer an improved apparatus for capturing images using a Simultaneous Localization and Mapping (SLAM) approach. SLAM requires finding non-consecutive frames showing a previously-visited area of the seafloor. This process is known as a loop-closure, and it allows for update of prior information about a given trajectory. It is often a non-trivial task even for neighboring lines in the lawnmower pattern due to small overlap, slightly varying camera speed and altitude, and/or relatively featureless areas. In some simulations there is no better solution than the brute force approach—trying to match each frame with each.
Without accurate positioning of the underwater system, even carefully planned surveys sometimes leave gaps or areas of insufficient non-consecutive overlap that introduce errors in resulting maps. To avoid such errors, some surveyors lay ropes or cables on the seafloor prior to data collection which significantly lengthens the time necessary for the survey. Even AUVs programmed to perform the survey mission can be thrown off course by a current not taken into account at the planning stage.
As shown in
One example uses an underwater motor 50 with an arm 70 attached by its center to the motor shaft such that the arm 70 rotates in a plane parallel to the seafloor in a direction 65. In some examples, one or more cameras 60 are attached to the arms 70. One or more, two or more, three or more, five or more, or ten or more cameras 60 or sensors may be utilized for detection of a region of interest. In some examples, the one or more cameras 60 are positioned facing vertically down. In some examples, the one or more cameras 60 are positioned at an angle with respect to the seafloor. When the arms 70 rotate, the cameras 60 acquire images of the circle with the diameter equal to the length of the arms 70 and the size of the motor 50.
When the motor drives forward moving in a straight-line direction 55 (linear motion, keeping constant altitude), the circle covered by the rotating cameras 60 also shifts covering new areas (
Structurally, a set of cameras 60 is provided that are configured to rotate in a direction 65 about a central motor 50. As the motor 50 drives in a forward linear direction 55, the set of cameras 60 rotates about the central motor 50.
In some examples, it may be desirable to produce 2.5D image reconstructions using structure from motion. Structure from motion photogrammetry may provide hyperscale landform models using images acquired from a range of digital cameras and optionally a network of ground control points. Structure from motion photogrammetry may provide point cloud data. Structure from motion may be useful in remote or rugged environments where terrestrial laser scanning is limited by equipment portability and airborne laser scanning is limited by terrain roughness causing loss of data and image foreshortening. Structure from motion may be applied in many settings such as rivers, badlands, sandy coastlines, fault zones, and coral reef settings. Structure from motion photogrammetry may be configured to provide detailed surface topography in unprecedented detail, multi-temporal data, elevation detection, or detection of position and volumetric changes providing details on earth surface processes, for example. Structure from motion may include using data acquired by a camera in free motion and performing postprocessing such that the resulting image shows a non-flat structure.
In some examples, the apparatus and methods disclosed herein can be utilized for measurement of underwater depth of lake or ocean floors. Cameras 60 may be configured for direct acquisition of bathymetry, or underwater depth of the ocean floor, including locating peaks and valleys and regions of flat terrain. For bathymetry or micro-bathymetry analysis, the one or more cameras 60 may be replaced with one or more time-of-flight (TOF) sensors, such as, for example a Hamamatsu S11963-01CR sensor. This device may simultaneously acquire a 160×120 array of distance measurements. The spectral response of the sensor (
The more than 100% coverage of the area of interest can be achieved for any reasonable camera footprint—its size requires only proper adjustment of rotational and linear speeds. Structurally, the set of cameras 60 may include 2, 3, 4, 5, 6, or more cameras. For a given set of cameras 60, if more cameras 60 are provided slower rotational speeds may be realizable. If fewer cameras 60 are provided, then a faster rotational speed would be required to capture images covering the same region.
The arm 70 with cameras 60 at the ends may be replaced with a solid disk that may be heavier than the arm 70 but has a number of advantages. First, the solid disk has a reduced drag than the arm 70 while rotating. Second, the solid disk allows for mounting more than two cameras 60 along its edge. The latter guarantees larger than 100% coverage with faster linear speed.
The motor may be handheld as the rotating arm 70 has properties similar to a gyroscope and once it starts to rotate, it tends to stay in the same plane of rotation. However, it may be mounted on a self-propelling platform that helps to keep the motor in a vertical position with little corrections from a diver and helps to maintain almost constant linear speed. The whole device may also be mounted on a vehicle, such as ROV or AUV.
One advantage of the proposed device is that, in some examples, a survey exploiting this device can be successfully performed by non-trained personnel. The proposed device eliminates the need to lay ropes or other markers on the seafloor. The proposed device eliminates the need to devise a mission plan for an AUV. When the surveyed area size exceeds the swath covered in one transect and there is still a need to employ the lawnmower pattern, there are less tough restrictions on the accuracy of the path in the opposite direction. More specifically, the proposed device is configured to compensate by matching non-sequential images as well as sequential images to form an accurate mosaic.
In the apparatus of the present disclosure, a relationship exists between linear speed of the motor, rotational speed of the arms 70 extending from the motor, field of view of the one or more cameras 60, length of the arms 70, or diameter of the disc, altitude of the camera 60 above the surface being imaged. For example, when the linear speed is low, rotational speed can also be low. However, for higher linear speeds, the rotational speed would need to be higher or more cameras 60 would need to be added to the system to cover the same area.
Coverage of the surveyed area using the apparatus and method described herein may depend on parameters including positional elevation above the area to be imaged, vertical field of view (FOV), rotational speed, and number of cameras. The vertical field of view of the camera FoVv (degrees) may be in the range of 30 degrees to 50 degrees, 35 degrees to 45 degrees, or 37 degrees to 42 degrees. The rotational speed vr may be in the range of 0.1 revolutions per second to 2 revolutions per second, 0.7 revolutions per second to 1.7 revolutions per second, or 1.0 revolutions per second to 1.5 revolutions per second. The number of cameras Nc may be in the range of 1 camera to 6 cameras. Variations in these ranges may also result in effective system performance.
Image or data acquisition would need to occur at a rate which is fast enough (i.e. a sufficiently short time period between sequential snapshots) to support a system configured to provide gap free image or data acquisition. For a given time period between sequential snapshots, the maximum linear speed of the device can be estimated as:
The above formula provides the upper limit of the linear speed that guarantees that there will be no gaps in detected area. Substituting typical values of parameters, maximum linear speed of the motor may lie in the range from 0.5 m/s to 112 m/s. Maximum linear speed for any given example may depend on hardware, water clarity, or other factors. Note that maximum linear speed does not depend on the length of the arm, the distal end of which supports and connects the one or more cameras. However, linear speed, which may be limited by potential drag, determines the width of the swath imaged by a single pass of the device. For a specific set of hardware and survey conditions, the maximum linear speed may be determined by simulation, which may also provide information about overlapping non-sequential frames.
Taking into account the frequency of snapshots would yield a more complex version of the formula for determining maximum linear speed.
To provide some insight into the dependence of area coverage on important parameters such as rotational speed, linear speed, size of area to be imaged, and snapshot period,
It is important to note that the system and methods described herein provide for high predictability of non-sequential overlap without the use of manual markers placed in a region to be imaged. The high image overlap provided by the system described herein allows for predictable guaranteed non-sequential overlap for future captured frames once the first non-sequential overlap has been identified.
Following global optimization utilizing SLAM and non-sequential image overlap, a full mosaic from all frames can be built. In the proof of concept, about 750 frames were captured.
Functionally, the apparatus described herein is configured to detect images using a central motor 50, a set of arms 70 extending from the central motor, and one or more rotating cameras 60 or sensors fixed to the set of arms 70. The apparatus 10 is configured to detect, or image, an area. The apparatus 10 is configured to provide improved image overlap of the predetermined region of interest as compared to conventional systems. The apparatus 10 is configured to minimize missing portions of the region to be mapped or analyzed. The apparatus 10 described herein is configured to provide improved image overlap and thereby provide a more complete and higher quality map of a predetermined region of interest.
In some examples, super-resolution techniques may be employed during image acquisition. Super-resolution techniques may enhance the resolution of the imaging system. For example, in some examples, a given sensor may use both optics and ultrasound for detection of a predetermined region of interest. In some examples, optical super resolution may be employed to transcend the diffraction limit of the system. Optical super resolution may involve substituting spatial-frequency bands, multiplexing spatial-frequency bands, use of multiple parameters within a traditional diffraction limit, and/or probing of a near-field electromagnetic disturbance.
In some examples, geometrical super resolution may be employed to enhance the resolution of digital imaging sensors. Geometrical super resolution may involve multi-exposure image noise reduction, single frame deblurring, and/or sub-pixel image localization. Multiple images of the same scene allow one to apply super resolution techniques that result in images with resolution higher than the original ones. In some examples, radar and sonar imaging may be enhanced using substance decomposition-based methods and compressed sensing-based algorithms. Super resolution techniques are configured generally to improve resolution of the resulting reconstructed map.
In some examples of the present disclosure, a computer is provided for processing received images and associated image locations. The computer may be configured to construct or update a map of an unknown environment.
Underwater, two approaches may be utilized which track underwater vehicles using acoustics to determine positioning. In the long baseline (LBL) approach several transponders are placed in an area of interest, or worksite, to be mapped. In some examples, a network of sea-floor mounted transponders may be used as reference points for navigation. Transponders may be placed around the perimeter of a worksite. The LBL technique enables matching the location of each image with high accuracy. Accuracy is generally better than 1 meter and can reach a few centimeters accuracy. Positional accuracy and position stability may be independent of water depth.
In the long baseline approach, the position of an AUV or ROV can be determined by acoustically measuring the distance from the AUV or ROV to three or more seafloor deployed baseline transponders. Range measurements may be supplemented by depth data from pressure sensors on the transponders. Range measurements may be used to triangulate the positioning of the underwater vehicle.
In some examples, a vehicle mounted interrogator (A) may send a signal, which is received by a set of baseline transponders (B, C, D). Vehicle mounted interrogator (A) may send a signal at a given frequency X and receive a signal at a given frequency Y1, Y2, and Y3, for example, from each of baseline transponders (B, C, D), respectively. A distance (d) and position of the vehicle from each baseline transponder may be calculated using time for each signal to reach the vehicle from each baseline transponder according to d=v*t, where velocity (v) is the speed of sound, approximately 1500 m/s. The speed of sound may vary depending on temperature, salinity, and even sound frequency. Calculated positions may be relative to the location of the baseline transponders. Positions may be converted to a geo-referenced coordinate system such as latitude/longitude or UTM (universal transverse Mercator) if geo-locations of the baseline stations are first established.
LBL may provide advantages over alternative positioning systems as alternative measurement systems, such as ultra-short baseline (USBL), use shorter baselines where range disturbances of a given amount can result in much larger position errors.
In some examples, the USBL approach may be used to track image position. Ultra-short baseline is an alternative method of underwater acoustic positioning. In USBL, a transceiver may be mounted to a transceiver on a pole under a ship and a transponder may be mounted on the seafloor or on a vehicle, such as an ROV or AUV. A computer may be used to calculate a position from the ranges and bearings measured by the transceiver.
An acoustic pulse may be transmitted by the transceiver and detected by the transponder, which then replies with an acoustic pulse of its own. The return pulse is detected by the transceiver aboard the ship. The time from the transmission of the initial acoustic pulse until the reply is detected is measured by the USBL system and is converted into a range.
To calculate subsea positioning, the USBL calculates both a range and an angle from the transceiver to the subsea transponder. The transceiver may contain an array of transducers for measuring received signal angle. A method called “phase-differencing” within the transducer array is used to calculate the direction to the subsea transponder.
Either the long baseline or ultra-short baseline approach for acoustic positioning may be used to supplement the system and methods described herein by providing position information for a given image. Positioning information from LBL is typically on the level of meters and from USBL is typically on the level of tens of centimeters. In particular, use of LBL or USBL approaches may be helpful for in cases where the proposed device is used with a boustrophodonic pattern to provide a rough alignment.
However, it is important to note that creation of a mosaic using the system and methods described herein is achievable with pixel level accuracy. Creation of a mosaic with millimeter or even sub-millimeter accuracy is achievable using the system and methods described herein. The system and methods described herein are configured to produce a seamless or near-seamless mosaic as a result of image processing techniques including bundle adjustment, non-sequential image overlap, global optimization, and utilizing the SLAM approach.