System and method of operation for remotely operated vehicles with superimposed 3D imagery

• US 10,424,119 B2
• Filed: 07/20/2017
• Issued: 09/24/2019
• Est. Priority Date: 11/09/2011
• Status: Active Grant

• Current Assignee: Ocean Infinity SA
• Original Assignee: Abyssal S.A.
• Inventors: Parente Da Silva, Manuel Alberto
• Primary Examiner(s): Nguyen, Tan Q
• Application Number:
  US15/654,974
• Publication Number:
  US 20170323487A1
• Time in Patent Office: 796 Days
• Field of Search: 701 2, 342 5
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The present invention provides a system and method of utilizing superimposed 3D imagery for remotely operated vehicles, namely 3D, reconstructed images of the environment of the ROV. In another aspect, it includes generating a virtual video of 3D elements in the operation environment, synchronizing the angle and position of the camera of a virtual video with the angle and position of a real camera, superimposing the virtual video and the real video from the real camera; superimposing these video feeds such that one is manipulated to show transparencies in areas of less interest, in order to show through the other video. It furthermore may include superimposing information, whether graphic, textual or both on to the hybrid virtual-real 3D imagery. The subject invention is also networked, such that the immersive visual interface described above is accessible to a plurality of users operating from a plurality of locations.

Claims

1. A system for exploration comprising:

a visualization engine, wherein said visualization engine further comprises;

a database module of 3D elements representing objects disposed in an operation environment;

a virtual video generating module for generating a virtual video feed incorporating said 3D elements; and

a superimposition module for superimposing said virtual video feed and a real video feed, said superimposed virtual and real video feeds comprising hybrid 3D imagery, wherein the superimposition module is configured to divide the virtual and real video feeds into a plurality of layers to permit the flattening and superimposition of the plurality of layers to produce hybrid 3D imagery that simulates spatial perception; and

a navigation interface configured to display said hybrid 3D imagery.

2. The system according to claim 1, wherein the superimposition module is configured to superimpose graphic information, textual information, or both, onto the hybrid 3D imagery.

3. The system according to claim 1, wherein said superimposing is based on a luminance threshold.

4. The system according to claim 1, wherein the system is further configured to define an actionable area, wherein entering, leaving, or remaining longer than a designated time in the actionable area triggers a system action.

5. The system according to claim 1, further comprising an external system configured to determine whether the system is working or is in a fail state.

6. The system according to claim 5, wherein the external system is configured to switch a monitor input from hybrid 3D imagery to said real video feed if the system is in a fail state.

7. The system according to claim 1, wherein the navigation interface comprises at least three networked monitors, wherein said monitors are arranged adjacent to one another such that the middle monitor displays superimposed virtual and real video feeds, while both side monitors display an expanded view of a field of operation.

8. The system according to claim 1, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.

9. The system according to claim 1, wherein said superimposition module is configured to modulate the transparency or opaqueness of a region of lesser interest in one of the virtual or real video feeds such that a corresponding region of the other video feed is more visible.

10. A method for exploration, comprising:

generating a virtual video feed of 3D elements representing objects disposed in an operation environment;

obtaining a real video feed; and

superimposing said virtual video feed and said real video feed, said superimposed virtual and real video feeds comprising hybrid 3D imagery, wherein superimposing comprises dividing the virtual and real video feeds into a plurality of layers to permit the flattening and superimposition of the plurality of layers to produce hybrid 3D imagery that simulates spatial perception.

11. The method according to claim 10, further comprising superimposing graphic information, textual information, or both, onto the hybrid 3D imagery.

12. The method according to claim 10, wherein superimposing further comprises the step of modulating the transparency or opaqueness of a region of lesser interest in one of the virtual or video feeds such that a corresponding region of the other video feed is more visible.

13. The method according to claim 12, wherein the step of modulating the transparency or opaqueness further comprises establishing a luminance threshold.

14. The method according to claim 10, further comprising synchronizing the angle and position of a video camera obtaining the real video feed, wherein synchronizing the angle and position of the video camera comprises the steps of:

calculating an angle between a heading of a remote operated vehicle and a direction of a field of view of the video camera;

calculating an angle between a vertical orientation of the remote operated vehicle and the direction of the video camera field of view; and

calculating an angle between the vehicle and a geographic horizon.

15. The method according to claim 10, wherein the step of superimposing further comprises modulating the amount of 2D or 3D information superimposed based on the position of a remote operated vehicle relative to objects disposed in the remotely operated vehicle'"'"'s operation environment.

16. The method according to claim 10, wherein the step of superimposing further comprises providing a minimap, said minimap defining a computer-generated graphic showing either or both cardinal points or a position of an object in 3D.

17. The method according to claim 10, comprising the step of configuring the operating system such that entering, leaving, or remaining longer than a designated time at a GPS-determined position triggers any one or combination of an alarm, notification, procedure change or task change.

18. The method according to claim 10, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.

19. The method according to claim 10, wherein superimposing further comprises the step of modulating the transparency or opaqueness of a region of lesser interest in one of the virtual or real video feeds such that a corresponding region of the other video feed is more visible.

20. A computer-readable medium including code for exploration, the code when executed operable to:

generate a virtual video feed of from 3D elements representing objects disposed in an operation environment;

obtain a real video feed; and

superimpose said virtual video feed and said real video feed by dividing the virtual and real video feeds into a plurality of layers to permit the flattening and superimposition of the plurality of layers to produce hybrid 3D imagery that simulates spatial perception.

21. The computer readable medium according to claim 20, wherein the code is further operable to:

calculate an angle between a heading of a remote operated vehicle and a direction of a field of view;

calculate an angle between a vertical orientation of the vehicle and the direction of the field of view; and

calculate an angle between the vehicle and a geographic horizon.

22. The computer readable medium according to claim 20, wherein the 3D elements are obtained from any of pre-recorded sonar data, live sonar data, programmed 3D objects, user-determined inputs, and bathymetry.

23. The computer readable medium according to claim 20, wherein superimposing further comprises the step of modulating the transparency or opaqueness of a region of lesser interest in one of the virtual video feed or video such that a corresponding region of the other video is more visible.