System and method for processing captured images

• US 10,726,616 B2
• Filed: 11/12/2019
• Issued: 07/28/2020
• Est. Priority Date: 06/17/2015
• Status: Active Grant

• Current Assignee: Rosemount Aerospace, Inc. (Rtx Corporation)
• Original Assignee: Rosemount Aerospace, Inc. (Rtx Corporation)
• Inventors: Ryde, Julian C., Ding, Xuchu
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US16/680,775
• Publication Number:
  US 20200082616A1
• Time in Patent Office: 259 Days
• Field of Search: None
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light

Abstract

An image processing system configured to process perceived images of an environment includes a central processing unit (CPU) including a memory storage device having stored thereon a computer model of the environment, at least one sensor configured and disposed to capture a perceived environment including at least one of visual images of the environment and range data to objects in the environment, and a rendering unit (RU) configured and disposed to render the computer model of the environment forming a rendered model of the environment. The image processing system compares the rendered model of the environment to the perceived environment to update the computer model of the environment.

Claims

1. A system for processing images captured by a moving platform comprising a steering system and a navigation system, the system comprising:

a central processing unit (CPU), said CPU being interconnected functionally via a system bus to;

at least one sensor configured and disposed to capture a perceived environment including at least one of visual images of the environment and range data to objects in the environment;

a graphics processing unit (GPU); and

at least one memory device thereupon stored a computer model of the environment and a set of instructions which, when executed by said CPU, causes the image processing system to;

capture a perceived environment through the at least one sensor, the perceived environment including visual images and range data;

render the computer model of the environment in the GPU forming a rendered model of the environment using image processing, the rendered model comprising a plurality of grids indicating one of a presence and an absence of the objects;

compare the rendered model of the environment with the perceived environment;

selectively update the computer model of the environment based on the perceived environment;

employ an iterative process to compute a position and orientation of the moving platform by performing an optimization of the perceived environment against the computer model of the environment, wherein the optimization comprises a direct numerical optimization on a per-point error metric; and

provide the computer model to the navigation system to control the steering system to direct the moving platform through the environment.

2. The system of claim 1, wherein the set of instructions which, when executed by said CPU, causes the image processing system to:

form at least one of a plurality of points, voxels, and textured polygon meshes that embody the perceived environment.

3. The system of claim 2, wherein each of the at least one of the plurality of points, voxels, and textured polygon meshes includes geographical positioning system (GPS) coordinates.

4. The system of claim 1, wherein the at least one sensor comprises a red, green, blue, and distance (RGBD) device.

5. The system of claim 1, wherein the at least one sensor comprises a first sensor configured to capture visual images of the environment, and a second sensor configured to determine range to objects in the environment.

6. The system according to claim 5, wherein the first sensor comprises a camera.

7. The system according to claim 5, wherein the second sensor comprises a LiDAR sensor.

8. A system for processing images captured by a moving platform comprising a steering system and a navigation system, the system comprising:

a central processing unit (CPU), said CPU being interconnected functionally via a system bus to;

at least one sensor configured and disposed to capture a perceived environment including at least one of visual images of the environment and range data to objects in the environment;

a graphics processing unit (GPU); and

at least one memory device thereupon stored a computer model of the environment and a set of instructions which, when executed by said CPU, causes the image processing system to;

capture a perceived environment through the at least one sensor, the perceived environment including visual images and range data;

render the computer model of the environment in the GPU forming a rendered model of the environment using image processing, the rendered model comprising a plurality of grids indicating one of a presence and an absence of the objects;

selectively update the computer model of the environment based on the perceived environment;

employ an iterative process to compute a position and orientation of the moving platform by performing an iterative closest point (ICP) optimization against the computer model of the environment, wherein the ICP optimization includes a point-to-point optimization or a point-to-plane optimization of the perceived environment; and

provide the computer model to the navigation system to control the steering system to direct the moving platform through the environment.

9. The system of claim 8, wherein the set of instructions which, when executed by said CPU, causes the image processing system to:

form at least one of a plurality of points, voxels, and textured polygon meshes that embody the perceived environment.

10. The system of claim 9, wherein each of the at least one of the plurality of points, voxels, and textured polygon meshes includes geographical positioning system (GPS) coordinates.

11. The system of claim 8, wherein the at least one sensor comprises a red, green, blue, and distance (RGBD) device.

12. The system of claim 8, wherein the at least one sensor comprises a first sensor configured to capture visual images of the environment, and a second sensor configured to determine range to objects in the environment.

13. The system according to claim 12, wherein the first sensor comprises a camera.

14. The system according to claim 12, wherein the second sensor comprises a LiDAR sensor.