Augmented reality sonar imagery systems and methods

• US 10,191,153 B2
• Filed: 02/27/2017
• Issued: 01/29/2019
• Est. Priority Date: 09/02/2014
• Status: Active Grant

• Current Assignee: Teledyne FLIR LLC (Teledyne Technologies Incorporated)
• Original Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Gatland, Christopher Daniel
• Primary Examiner(s): Pihulic, Daniel
• Application Number:
  US15/443,836
• Publication Number:
  US 20170168159A1
• Time in Patent Office: 701 Days
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/89 for mapping or imaging
• CPC Class Codes: G01S 15/96 for locating fish
• CPC Class Codes: G01S 7/6263 in which different colours are used
• CPC Class Codes: G01S 7/6281 Composite displays, e.g. split-screen, multiple images

Abstract

Techniques are disclosed for systems and methods to provide augmented reality sonar imagery for mobile structures. An augmented reality sonar imagery system includes a portable imaging device, a sonar transducer assembly, and a logic device in communication with the sonar transducer assembly and the portable imaging device. The sonar transducer assembly is adapted to be mounted to a mobile structure and placed in a body of water, and the portable imaging device includes a display and an imager position and/or orientation sensor. The logic device is configured to determine a waterline of the body of water relative to a field of view of the display and render sonar data in a portion of the field of view that extends below the waterline. Subsequent user input and/or the sonar data may be used to adjust a steering actuator, a propulsion system thrust, and/or other operational systems.

Claims

1. A system comprising:

a portable imaging device comprising a display and an imager position and/or orientation sensor (IPOS);

a sonar transducer assembly adapted to be mounted to a mobile structure and placed in a body of water; and

a logic device in communication with the sonar transducer assembly and the portable imaging device, wherein the logic device is configured to;

determine a waterline of the body of water relative to a field of view (FOV) of the display; and

render sonar data in a portion of the FOV that extends below the waterline.

2. The system of claim 1, wherein the logic device is configured to:

determine the waterline using image data, position data, and/or orientation data provided by the portable imaging device and/or IPOS.

3. The system of claim 1, wherein the logic device is configured to:

render image data provided by the portable imaging device in another portion of the FOV that extends above the waterline.

4. The system of claim 1, wherein:

the sonar transducer assembly comprises a sonar position and/or orientation sensor (SPOS); and

the logic device is configured to render the sonar data in the portion of the FOV that extends below the waterline using position data and/or orientation data provided by the SPOS.

5. The system of claim 1, wherein:

the sonar transducer assembly comprises a sonar position and/or orientation sensor (SPOS) and a sonar actuator configured to adjust a position and/or orientation of the sonar transducer assembly; and

the logic device is configured use the SPOS and the sonar actuator to ensonify at least a subset of the portion of the FOV that extends below the waterline.

6. The system of claim 1, further comprising a second portable imaging device in communication with the logic device, wherein the logic device is configured to:

determine a second waterline relative to a second FOV of a second display of the second portable imaging device; and

render sonar data in a portion of the second FOV that extends below the second waterline.

7. The system of claim 1, wherein:

the mobile structure comprises a watercraft;

the logic device is configured to determine the portion of the FOV that extends below the waterline overlaps with a view of the mobile structure; and

the rendering the sonar data comprises blending image data of the mobile structure with the sonar data in the overlapping portion of the FOV and rendering the blended data in the overlapping portion.

8. The system of claim 1, wherein:

the portable imaging device is adapted to be held or worn by a user of the system while the user views the display.

9. The system of claim 1, wherein:

the portable imaging device comprises one or more image modules adapted to be mounted to the mobile structure above the waterline;

the display is adapted to be held or worn by a user of the system while the user views the display;

the IPOS comprises one or more module IPOSs configured to measure positions and/or orientations of the image modules and a display IPOS configured to measure a position and/or orientation of the display; and

the logic device is configured to;

determine the waterline using image data, position data, and/or orientation data provided by the image modules, module IPOSs, and/or display IPOS; and

render image data provided by the image modules in another portion of the FOV that extends above the waterline.

10. The system of claim 1, wherein:

the portable imaging device comprises a visible spectrum image module and an infrared image module; and

the logic device is configured to;

generate combined image data from visible spectrum image data and infrared image data provided by the portable imaging device; and

render the combined image data in at least another portion of the FOV that extends above the waterline.

11. The system of claim 1, wherein the rendering the sonar data comprises:

rendering substantially real time sonar data in color and prior-acquired sonar data in greyscale;

reducing a chrominance level of the sonar data as the sonar data ages;

determining one or more surfaces from the sonar data and mapping orientations of the surfaces to a color map and/or an intensity map; and

/or

rendering a portion of a survey map stored in a memory coupled to the logic device.

12. A method comprising:

determining a waterline of a body of water relative to a field of view (FOV) of a display of a portable imaging device, wherein the portable imaging device comprises an imager position and/or orientation sensor (IPOS);

receiving sonar data from a sonar transducer assembly adapted to be mounted to a mobile structure and placed in the body of water; and

rendering the sonar data in a portion of the FOV that extends below the waterline.

13. The method of claim 12, further comprising:

determining the waterline using image data, position data, and/or orientation data provided by the portable imaging device and/or IPOS; and

rendering image data provided by the portable imaging device in another portion of the FOV that extends above the waterline.

14. The method of claim 12, wherein:

the sonar transducer assembly comprises a sonar position and/or orientation sensor (SPOS); and

the method comprises rendering the sonar data in the portion of the FOV that extends below the waterline using position data and/or orientation data provided by the SPOS.

15. The method of claim 12, wherein:

the sonar transducer assembly comprises a sonar position and/or orientation sensor (SPOS) and a sonar actuator configured to adjust a position and/or orientation of the sonar transducer assembly; and

the method comprises using the SPOS and the sonar actuator to ensonify at least a subset of the portion of the FOV that extends below the waterline.

16. The method of claim 12, further comprising:

determining a second waterline relative to a second FOV of a second display of a second portable imaging device; and

rendering sonar data in a portion of the second FOV that extends below the second waterline.

17. The method of claim 12, wherein:

the method comprises determining the portion of the FOV that extends below the waterline overlaps with a view of the mobile structure; and

the rendering the sonar data comprises blending image data of the mobile structure with the sonar data in the overlapping portion of the FOV and rendering the blended data in the overlapping portion.

18. The method of claim 12, wherein:

the portable imaging device comprises one or more image modules adapted to be mounted to the mobile structure above the waterline;

the display is adapted to be held or worn by a user of the system while the user views the display;

the IPOS comprises one or more module IPOSs configured to measure positions and/or orientations of the image modules and a display IPOS configured to measure a position and/or orientation of the display; and

the method comprises;

determining the waterline using image data, position data, and/or orientation data provided by the image modules, module IPOSs, and/or display IPOS; and

rendering image data provided by the image modules in another portion of the FOV that extends above the waterline.

19. The method of claim 12, wherein:

the portable imaging device comprises a visible spectrum image module and an infrared image module; and

the method comprises;

generating combined image data from visible spectrum image data and infrared image data provided by the portable imaging device; and

rendering the combined image data in at least another portion of the FOV that extends above the waterline.

20. The method of claim 12, wherein the rendering the sonar data comprises:

rendering substantially real time sonar data in color and prior-acquired sonar data in greyscale;

reducing a chrominance level of the sonar data as the sonar data ages;

determining one or more surfaces from the sonar data and mapping orientations of the surfaces to a color map and/or an intensity map; and

/or

rendering a portion of a survey map stored in a memory coupled to the logic device.