ENHANCED SITUATIONAL AWARENESS AND TARGETING (eSAT) SYSTEM

• US 20120019522A1
• Filed: 07/14/2011
• Published: 01/26/2012
• Est. Priority Date: 07/25/2010
• Status: Abandoned Application

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Wolske, Jeff S., Lawrence, Robert J., Prostejovsky, Eric S., O'Neal, William B.
• Application Number:
  US13/183,155
• Publication Number:
  US 20120019522A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: F41G 3/02 using an independent line of sight
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation

Abstract

eSAT pushes 3D scene awareness and targeting to forward positioned mobile operators and their handheld devices in environments such as found in military theaters of operation, border control and enforcement, police operations, search & rescue and large commercial industrial operations. A host computer hosts a 3D model of a scene and dynamically captures and transmits a windowed portion of the visual representation of that 3D model over a wireless network to the mobile operators'"'"' handheld devices. The windowed portion of the visual representation is streamed directly to the operators'"'"' handheld device displays. The mobile operators may interact with and control the 3D model via the wireless network. The host computer may synthesize the visual representation of the 3D model with live feeds from one or more of the handheld devices or other assets to improve situational awareness. Either the mobile operators or host operator can make point selections on the visual representation to extract geo-coordinates from the 3D model as a set of target coordinates.

Claims

1. A method of providing enhanced situational awareness to a host operator and one or more forward positioned mobile operators, comprising:

providing a host computer with a three-dimensional (3D) model rendered from images of a scene, said 3D model capable of geometric manipulation from different viewpoints;

displaying a visual representation of the 3D model from a specified viewpoint on a host computer display, said host computer configured to allow host operator manipulation of the 3D model to change the viewpoint;

dynamically capturing a first windowed portion of the visual representation of the 3D model at a given refresh rate; and

streaming the first windowed portion of the visual representation over a wireless network to the handheld devices of one or more mobile operators forward positioned in the vicinity of the scene, said handheld devices configured to allow mobile operator selection to stream the first windowed portion of the visual representation to a handheld device display.

2. The method of claim 1, further comprising:

from a platform of known geo-coordinates, laser position sensing at least three points from the scene to extract geo-coordinates for each of the at least three points in the 3D model; and

using the extracted geo-coordinates, geo-rectifying the 3D model to geo-coordinates to geo-register the 3D model in scale, translation and rotation and displaying the visual representation of the 3D model from geo-accurate viewpoints.

3. The method of claim 2, wherein the platform comprises an airborne asset with an overhead view of the scene or a terrestrial asset with a side view of the scene.

4. The method of claim 1, further comprising:

receiving images of the scene from forward positioned mobile assets; and

processing the received images into 3D model data at the host computer and integrating the data to update the 3D model in time, spatial extent or resolution.

5. The method of claim 4, wherein the forward positioned mobile units comprise one or more of the mobile operators'"'"' handheld devices, said handheld devices configured to capture still or moving images and transmit the images over the wireless network to the host computer.

6. The method of claim 1, wherein the step of providing the 3D model, comprises:

flying a forward positioned unmanned aerial vehicle (UAV) with a camera above the scene at diverse viewpoints to record and provide images directly to the host computer to render the images into the 3D model.

7. The method of claim 6, further comprising:

slaving the viewpoint of the visual representation of the 3D model to the current viewpoint of the UAV.

8. The method of claim 1, further comprising:

displaying a second visual representation of the same 3D model from a different viewpoint on the host computer display;

dynamically capturing a second windowed portion of the second visual representation of the 3D model at a given refresh rate; and

streaming the second windowed portion of the second visual representation over a wireless network to the handheld devices of one or more mobile operators forward positioned in the vicinity of the scene, said handheld devices configured to allow mobile operator selection of the second window to stream the second windowed visual representation in real-time to the handheld device display.

9. The method of claim 1, further comprising:

dynamically capturing a second windowed portion of the visual representation from the specified viewpoint at the given refresh rate, said second windowed portion of the visual representation different from said first windowed portion of the visual representation; and

streaming the second windowed portion of the visual representation over the wireless network to the same or different one or more of the handheld devices of the mobile operators.

10. The method of claim 1, further comprising:

said one or more forward positioned mobile operators capturing live feeds of images of the scene and transmitting the live feeds over the wireless network to the host computer; and

displaying the one or more live feeds in one or more additional windows on the host computer display in conjunction with the visual representation of the 3D model.

11. The method of claim 10, further comprising:

slaving the viewpoint of the visual representation of the 3D model to the current viewpoint of one of the live feeds.

12. The method of claim 11, further comprising:

embedding the display of the live feed to which the visual representation is slaved within the visual representation.

13. The method of claim 10, further comprising:

obtaining the geo-coordinates of at least each of the forward positioned mobile operators'"'"' handheld devices transmitting the live feeds; and

displaying a geo-positioned marker for each of the geo-located mobile operators within the visual representation of the 3D model.

14. The method of claim 1, further comprising:

host operator selection of a single point on the visual representation of the 3D model to extract geo-coordinates from the 3D model; and

transmitting the geo-coordinates of said point as a set of target coordinates to deploy an asset to the target coordinates.

15. The method of claim 1, further comprising:

one or more said forward positioned mobile operators generating a 3D model manipulation command from their handheld devices based on the streamed first windowed portion of the visual representation of the 3D model;

transmitting the one or more model manipulation commands over the wireless network to the host computer;

said host computer executing the one or more model manipulation commands on the 3D model to display visual representations of the 3D model in accordance with the commands; and

streaming the first windowed portion of the visual representation back to at least the requesting forward mobile operator.

16. The method of claim 15, wherein said host computer executes multiple model manipulation commands in a time sequence and streams the first windowed portions of the visual representations back to at least the requesting forward mobile operator sequentially.

17. The method of claim 15, wherein said host computer executes multiple model manipulation commands in parallel displaying different visual representations of the model in different windows on the host computer display and streams the respective first windowed portions of the visual representations back to the respective requesting forward mobile operators concurrently.

18. The method of claim 15, wherein icons for one or more host computer applications are dynamically captured by the first windowed portion of the visual representation streamed to the handheld displays, further comprising:

mobile operator selection of an icon on the handheld display to generate an application command;

transmitting the application command over the wireless network to the host computer;

said host computer executing the application command to update the visual representation; and

streaming the first windowed portion of the visual representation back to at least the forward mobile operator that generated the application command.

19. The method of claim 15, further comprising:

capturing and displaying a live feed of images from a mobile operator'"'"'s current viewpoint on the handheld device display; and

transmitting the model manipulation command in accordance with the current viewpoint so that the visual representation of the 3D model streamed back to the mobile operator'"'"'s handheld device is slaved to the operator'"'"'s current viewpoint.

20. The method of claim 1, further comprising:

flying an unmanned aerial vehicle (UAV) above the scene at a given viewpoint to capture and transmit still or moving images to the host computer; and

slaving the viewpoint of the UAV to the current viewpoint of the 3D model in response to commands from the host operator or a mobile operator via the wireless network.

21. The method of claim 1, further comprising:

one or more said forward position mobile operators selecting a point on the displayed first windowed portion of the visual representation to generate a target point selection command;

transmitting the one or more target point selection commands over the wireless network to the host computer;

said host computer matching the one or more target point selection commands to the 3D model to extract a set of target coordinates; and

said host computer transmitting the target coordinates to deploy an asset to the target coordinates.

22. The method of claim 21, wherein the host computer highlights the target coordinates on the first windowed portion of visual representation that is streamed back to the requesting forward mobile operator with a confirmation prompt.

23. The method of claim 1, further comprising:

from either the forward positioned host computer or the mobile operator'"'"'s handheld device, capturing a live feed of images of the scene in a global positioning satellite (GPS) signal-denied environment; and

correlating the live feed to the 3D model to estimate geo-coordinates of the host computer or mobile operator

24. The method of claim 23, further comprising displaying the live feed on the host computer display or handheld display and slaving the viewpoint of the visual representation of the 3D model to that of the live feed.

25. A method of providing enhanced situational awareness to a host operator and one or more forward positioned mobile operators, comprising:

providing a host computer with a three-dimensional (3D) model rendered from images of a scene capable of geometric manipulation from different viewpoints;

geo-rectifying the 3D model to geo-coordinates to geo-register the 3D model in scale, translation and rotation;

displaying a visual representation of the 3D model from a specified geo-accurate viewpoint on a host computer display, said host computer configured to allow host operator manipulation of the 3D model to change the viewpoint;

placing a first window on the host computer display, said host computer dynamically capturing a first windowed portion of the visual representation at a given refresh rate;

host computer streaming the first windowed portion of the visual representation over a wireless network to the handheld devices of one or more mobile operators forward positioned in the vicinity of the scene, said handheld devices configured to allow mobile operator to stream the first windowed portion of the visual representation to a handheld device display; and

mobile operator interaction via the wireless network with the 3D model to change the viewpoint of the rendered 3D model.

26. The method of claim 25, further comprising:

mobile operator interaction via the wireless network to select a point target on the 3D model; and

said host computer matching the selected point target to the 3D model to extract a set of target geo-coordinates.

27. The method of claim 25, further comprising:

capturing and displaying still or moving images from a mobile operator'"'"'s current viewpoint on the handheld device display; and

changing the viewpoint of the rendered 3D model to slave the visual representation to the current viewpoint of the handheld device display.

28. The method of claim 25, further comprising:

flying an unmanned aerial vehicle (UAV) above the scene at a given viewpoint to capture and transmit still or moving images to the host computer; and

slaving the viewpoint of the UAV to commands from the mobile operator and current viewpoint of the 3D model.

29. A method of providing enhanced situational awareness to a host operator and one or more forward positioned mobile operators, comprising:

providing a host computer with three-dimensional (3D) model rendered from 2D images of a scene capable of geometric manipulation from different viewpoints;

geo-rectifying the 3D model to geo-coordinates to geo-register the 3D model in scale, translation and rotation;

displaying a visual representation of the 3D model from a specified geo-accurate viewpoint on a host computer display, said host computer configured to allow host operator manipulation of the 3D model to change the viewpoint;

placement of a first window on the host computer display, said host computer dynamically capturing a first windowed portion of the visual representation at a given refresh rate;

host computer streaming the first windowed portion of the visual representation over a wireless network to the handheld devices of one or more mobile operators forward positioned in the vicinity of the scene, said handheld devices configured to allow mobile operator selection to stream the first windowed portion of the visual representation to a handheld device display;

said one or more forward positioned mobile operators capturing live feeds of images of the scene and transmitting the live feeds and their geo-coordinates over the wireless network to the host computer;

displaying the one or more live feeds in one or more additional windows on the host computer display in conjunction with the visual representation of the 3D model; and

displaying a geo-positioned icon for each of the geo-located mobile operators within the visual representation of the 3D model.

30. The method of claim 29, further comprising:

slaving the viewpoint of the visual representation of the 3D model to the current viewpoint of one of the live feeds.

31. The method of claim 29, further comprising:

host operator selection of a single point on the visual representation of the 3D model to extract geo-coordinates from the 3D model; and

transmitting the geo-coordinates of said point as a set of target coordinates to deploy an asset to the target coordinates.

32. A system for providing enhanced situational awareness to a host operator and one or more forward positioned mobile operators, comprising:

a camera for recording images from diverse viewpoints about a scene;

a computer for rendering the images into a three-dimensional (3D) model capable of geometric manipulation from different viewpoints;

a wireless network;

a host computer for displaying a visual representation of the 3D model from a specified viewpoint on its display, said host computer configured to allow host operator manipulation of the 3D model to change the viewpoint and placement of a first window on the host computer display, said host computer dynamically capturing a first windowed portion of the visual representation of the 3D model at a given refresh rate and streaming the first windowed portion of the visual representation over the wireless network;

a plurality of handheld devices, each said device comprising a display configured to allow mobile operator selection of a first window to stream the first windowed portion of the visual representation, a camera to capture a live feed of images of the modeled scene for presentation on the handheld display and transmission via the wireless network to the host computer via the wireless network for display in conjunction with the visual representation of the 3D model, a global positioning system (GPS) receiver for determining geo-coordinates of the handheld device and transmitting the geo-coordinates via the wireless network to the host computer for displaying a geo-positioned icon of the mobile operator within the visual representation of the 3D model and an application configured to support mobile operator interaction with the 3D model via the wireless network to change the viewpoint.

33. The system of claim 32, wherein the host computer slaves the viewpoint of the 3D model to the current viewpoint of the live feed from one of the handheld devices.

34. The system of claim 32, wherein the host computer is configured to allow host operator selection of a point on the visual representation, said host computer matching the selected point to the 3D model to extract a set of target coordinates and transmitting the target coordinates to deploy an asset to the target coordinates.

35. The system of claim 34, wherein the handheld displays are configured to allow mobile operator selection of a point on the visual display, said handheld devices transmitting the selected point to the host computer to match the selected point to the 3D model to extract a set of target coordinates and transmit the target coordinates to deploy an asset to the target coordinates.

36. The system of claim 32, further comprising:

an unmanned aerial vehicle (UAV), said UAV flown above the scene at a given viewpoint to transmit a live feed of images to the host computer, said viewpoint of the UAV slaved to the current viewpoint of the 3D model in response to commands from the host operator or a mobile operator via the wireless network.