System and method of operation for remotely operated vehicles with superimposed 3D imagery

• US 9,195,231 B2
• Filed: 11/08/2012
• Issued: 11/24/2015
• Est. Priority Date: 11/09/2011
• Status: Active Grant

• Current Assignee: Ocean Infinity SA
• Original Assignee: Abyssal S.A.
• Inventors: Parente Da Silva, Manuel Alberto
• Primary Examiner(s): NGUYEN, TAN QUANG
• Application Number:
  US14/357,100
• Publication Number:
  US 20140316611A1
• Time in Patent Office: 1,111 Days
• Field of Search: 701/2, 701/23, 701/25, 701/28, 701/21, 345/427, 345/633
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The present invention provides a system and method of utilizing superimposed 3D imagery for remotely operated vehicles, namely 3D, reconstructed images of the environment of the ROV. In another aspect, it includes generating a virtual video of 3D elements in the operation environment, synchronizing the angle and position of the camera of a virtual video with the angle and position of a real camera, superimposing the virtual video and the real video from the real camera; superimposing these video feeds such that one is manipulated to show transparencies in areas of less interest, in order to show through the other video. It furthermore may include superimposing information, whether graphic, textual or both on to the hybrid virtual-real 3D imagery. The subject invention is also networked, such that the immersive visual interface described above is accessible to a plurality of users operating from a plurality of locations.

Claims

1. A system for exploration comprising:

a remote operated vehicle (ROV), wherein said ROV comprises a video camera for acquiring a live video feed;

a networked operating system comprising a visualization engine, wherein said visualization engine further comprises;

a database module of 3D elements representing objects disposed in an operation environment of said remote operated vehicle;

a virtual video generating module for generating a virtual video feed incorporating said 3D elements;

a synchronizing module for synchronizing an angle and position of a virtual camera with an angle and position of the video camera, wherein the virtual camera defines a field of view for the virtual video feed, wherein the field of view is preferably between about 45 and 144 degrees from a central point of view; and

a superimposition module for superimposing said virtual video feed and said live video feed, wherein said superimposition module is configured to modulate the transparency or opaqueness of a region of lesser interest in one of the virtual or live video feeds such that a corresponding region of the other video feed is more visible, said superimposed virtual and live video feeds comprising hybrid 3D imagery; and

a navigation interface configured to display said hybrid 3D imagery, said navigating interface comprising at least one networked monitor.

2. The system according to claim 1, wherein the superimposition module is configured to superimpose graphic information, textual information, or both, onto the hybrid 3D imagery.

3. The system according to claim 1, wherein said superimposing is based on a luminance threshold in Red-Green-Blue hexadecimal format, wherein luminance is set to values between 0-0-0 and 255-255-255, and preferably between 0-0-0 and 40-40-40.

4. The system according to claim 1, wherein the operating system is capable of sharing data with a plurality of remote monitors, said plurality comprising up to 12 remote monitors, and preferably between 3 and 9 remote monitors.

5. The system according to claim 1, further comprising an external system configured to determine whether the system is working or is in a fail state.

6. The system according to claim 5, wherein the external system is configured to switch the monitor input from hybrid 3D imagery to said live video feed if the system is in a fail state.

7. The system according to claim 1, wherein the navigation interface comprises at least three networked monitors, wherein said monitors are arranged adjacent to one another such that the middle monitor displays superimposed virtual and live video feeds, while both side monitors display an expanded view of a field of operation.

8. The system according to claim 1, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.

9. A method for exploration, comprising:

storing 3D elements in a database module, said 3D elements representing objects disposed in the remotely operated vehicle'"'"'s operation environment;

generating a virtual video feed of said 3D elements;

obtaining a live video feed using a video camera;

synchronizing an angle and position of a virtual camera with an angle and position of the video camera mounted to the remotely operated vehicle, wherein the virtual camera defines a field of view for the virtual video feed; and

superimposing said virtual video feed and said live video feed on a video screen, wherein superimposing further comprises the step of modulating the transparency or opaqueness of a region of lesser interest in one of the virtual or live video feeds such that a corresponding region of the other video feed is more visible, said superimposed virtual and live video feeds comprising hybrid 3D imagery.

10. The method according to claim 9, further comprising superimposing graphic information, textual information, or both, onto the hybrid 3D imagery.

11. The method according to claim 9, wherein the step of modulating the transparency or opaqueness further comprises establishing a luminance threshold.

12. The method according to claim 11, wherein establishing the luminance threshold comprises maintaining a background of the virtual video feed at a higher transparency than the background of the real video feed.

13. The method according to claim 9, wherein synchronizing the angle and position of the video camera comprises the steps of:

calculating an angle between a heading of the vehicle and a direction of a field of view of the video camera;

calculating an angle between a vertical orientation of the vehicle and the direction of the video camera field of view; and

calculating an angle between the vehicle and a geographic horizon.

14. The method according to claim 9, wherein the step of superimposing further comprises modulating the amount of 2D or 3D information superimposed onto the video screen based on the position of the ROV relative to objects disposed in the remotely operated vehicle'"'"'s operation environment.

15. The method according to claim 9, wherein the step of superimposing further comprises providing a minimap, said minimap defining a computer-generated graphic showing either or both cardinal points or a position of an object in 3D.

16. The method according to claim 9, comprising the step of configuring the operating system such that entering, leaving, or remaining longer than a designated time at a GPS-determined position triggers any one or combination of an alarm, notification, procedure change or task change.

17. The method according to claim 9, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.

18. A computer-readable medium including code for exploration, the code when executed operable to:

store 3D elements in a database module, said 3D elements representing objects disposed in the remotely operated vehicle'"'"'s operation environment;

generate a virtual video feed of said 3D elements;

obtain a live video feed using a video camera;

synchronize an angle and position of a virtual camera with an angle and position of the video camera mounted to the remotely operated vehicle, wherein the virtual camera defines a field of view for the virtual video feed; and

superimpose said virtual video feed and said live video feed on a video screen, wherein superimposing further comprises the step of modulating the transparency or opaqueness of a region of lesser interest in one of the virtual or live video feeds such that a corresponding region of the other video feed is more visible, said superimposed virtual and live video feeds comprising hybrid 3D imagery.

19. The computer readable medium according to claim 18, wherein the code for synchronizing the angle and position of the video camera is further operable to:

calculate an angle between a heading of the vehicle and a direction of a field of view of the video camera;

calculate an angle between a vertical orientation of the vehicle and the direction of the video camera field of view; and

calculate an angle between the vehicle and a geographic horizon.

20. The computer readable medium according to claim 18, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.