GROUP OPTIMIZATION DEPTH INFORMATION METHOD AND SYSTEM FOR CONSTRUCTING A 3D FEATURE MAP

• US 20190244378A1
• Filed: 02/08/2018
• Published: 08/08/2019
• Est. Priority Date: 02/08/2018
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Dong, Haiwei, Ren, Yuan, Golestan Irani, Keyvan
• Granted Patent: US 10,643,342 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A group optimization method for constructing a 3D feature map is disclosed. In one embodiment, the method comprises determining correspondence information for a plurality of environmental features for each image in a group of images in which a respective environmental feature is present and relative position and alignment of each camera. Depth information is determined for each environmental feature in the plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on the correspondence information of the environmental features. Group optimized depth information is determined for each environmental feature in the plurality of environmental features using the determined depth information of each respective environmental feature.

Claims

1. A computer vision system, comprising:

a processor;

a plurality of cameras coupled to the processor, wherein the plurality of cameras are located about a host object, the plurality of cameras comprising at least two front facing cameras having a front facing view of an environment in which the object is located, at least one left side facing camera having a left side facing view of the environment in which the object is located, at least one right side facing camera having a right side facing view of the environment in which the object is located, and at least one rear facing camera having a rear facing view of the environment in which the object is located;

a memory coupled to the processor, the memory tangibly storing thereon executable instructions that, when executed by the processor, cause the computer vision system to;

select a group of images from images captured by the plurality of cameras, the group of images including one captured image for each camera in the plurality of cameras, which were captured at a common time;

perform image distortion compensation on each image in the group of images based on intrinsic parameters of the respective camera that captured the image;

rectify each image in the group of images so that all images in the group of images have a common image plane;

determine correspondence information for a plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on a relative position and alignment of each camera in the plurality of cameras;

determine depth information for each environmental feature in the plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on the correspondence information of the environmental features; and

determine group optimized depth information for each environmental feature in the plurality of environmental features using the determined depth information of each respective environmental feature.

2. The computer vision system of claim 1, wherein the executable instructions, when executed by the processor, cause the computer vision system to generate a three-dimensional (3D) feature map of the environment in which the object is located based on the group optimized depth information of each environmental feature in the plurality of environmental features.

3. The computer vision system of claim 2, wherein the executable instructions, when executed by the processor, cause the computer vision system to display the 3D feature map of the environment in which the object is located on a display.

4. The computer vision system of claim 2, wherein the executable instructions, when executed by the at least one processor, cause the computer vision system to output the 3D feature map of the environment in which the object is located to a vehicle driver assistance system or autonomous vehicle system.

5. The computer vision system of claim 1, wherein the executable instructions, when executed by the processor, cause the computer vision system to calculate a maximum likelihood estimation in accordance with a noise distribution model for each environmental feature in each image in which the environmental feature appears.

6. The computer vision system of claim 5, wherein the executable instructions, when executed by the processor, cause the computer vision system to output the maximum likelihood estimations for each environmental feature to a vehicle driver assistance system or autonomous vehicle system.

7. The computer vision system of claim 1, wherein the executable instructions, when executed by the processor, cause the computer vision system to, when selecting the group of images from the captured images, one captured image for each camera in the plurality of cameras in accordance with a time stamp of each image such that images in the group of images were captured at a common time.

8. The computer vision system of claim 1, wherein each camera has a different field of view, and wherein adjacent cameras have at least partially overlapping fields of view.

9. The computer vision system of claim 1, wherein the correspondence information defines a series of locations in two dimensional (2D) space defining coordinate locations corresponding to a respective environmental feature for each image in which the respective environmental feature is present.

10. The computer vision system of claim 1, wherein the depth information and group optimized depth information for each environmental feature comprises a series of locations in 3D space representing the respective environmental feature.

11. The computer vision system of claim 1, wherein the depth information and group optimized depth information for each environmental feature consists of a point location in 3D space representing a center or nearest point of the respective environmental feature.

12. The computer vision system of claim 1, wherein for a respective location in the correspondence information for a respective environmental feature, a group optimized depth is determined, wherein the group optimized depth is defined by a vector x defined according to the following equation:

13. The computer vision system of claim 1, wherein for a respective location in the correspondence information for a respective environmental feature, a group optimized depth is determined, wherein the group optimized depth is determined by:

calculating an observation error, f, for each camera in which a respective environmental feature is observed accordance with the following equation

f_i=√

{square root over ((x_o−

x_i)²+(y_o−

y_i)²+(z_o−

z_i)²)}−

r_i²

wherein i refers the i^th camera in which a respective environmental is observed,

wherein x_i, y_i, z_i denotes the center location of the i^th camera, wherein i=1 . . . n, wherein n is the number of cameras in which the respective environmental feature is observed, x_o, y_o, z_o denotes a 3D location of the environmental feature corresponding to the respective location in the correspondence information, and wherein r_i denotes the distance from the center location of i^th camera 1 to n to the respective intersection point x_o, y_o, z_o;

calculating a location of the intersection point P using iteration in accordance with the following equation

P_k+1=P_k−

(J_k^TJ_k)<sup>−

1</sup>J_k^Tf_k

wherein P_k and P_k+1 are the calculated locations of the intersection point P for the k^th and k+1^th iteration,

wherein P is defined as

14. A method of generating a three-dimensional (3D) feature map, comprising:

capturing images from each of a plurality of cameras, wherein the cameras are located about a host object, the plurality of cameras comprising at least two front facing cameras having a front facing view of an environment in which the object is located, at least one left side facing camera having a left side facing view of the environment in which the object is located, at least one right side facing camera having a right side facing view of the environment in which the object is located, and at least one rear facing camera having a rear facing view of the environment in which the object is located;

selecting a group of images from the captured images, one captured image for each camera in the plurality of cameras, which were captured at a common time;

performing image distortion compensation on each image in the group of images based on intrinsic parameters of the respective camera that captured the image;

rectifying each image in the group of images so that all images in the group of images have a common image plane;

determining correspondence information for a plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on a relative position and alignment of each camera in the plurality of cameras;

determining depth information for each environmental feature in the plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on the correspondence information of the environmental features; and

determining group optimized depth information for each environmental feature in the plurality of environmental features using the determined depth information of each respective environmental feature.

15. The method of claim 14, wherein for a respective location in the correspondence information for a respective environmental feature, a group optimized depth is determined, wherein the group optimized depth is defined by a vector x defined according to the following equation:

16. The method of claim 14, wherein for a respective location in the correspondence information for a respective environmental feature, a group optimized depth is determined, wherein the group optimized depth is determined by:

calculating an observation error, f, for each camera in which a respective environmental feature is observed accordance with the following equation

f_i=√

{square root over ((x_o−

x_i)²+(y_o−

y_i)²+(z_o−

z_i)²)}−

r_i²

wherein i refers the i^th camera in which a respective environmental is observed,

wherein x_i, y_i, z_i denotes the center location of the i^th camera, wherein i=1 . . . n, wherein n is the number of cameras in which the respective environmental feature is observed, x_o, y_o, z_o denotes a 3D location of the environmental feature corresponding to the respective location in the correspondence information, and wherein r_i denotes the distance from the center location of i^th camera 1 to n to the respective intersection point x_o, y_o, z_o;

calculating a location of the intersection point P using iteration in accordance with the following equation

P_k+1=P_k−

(J_k^TJ_k)<sup>−

1</sup>J_k^Tf_k

wherein P_k and P_k+1 are the calculated locations of the intersection point P for the k^th and k+1^th iteration,

wherein P is defined as

17. The method of claim 14, further comprising:

generating a three-dimensional (3D) feature map of the environment in which the object is located based on the group optimized position of each environmental feature in the plurality of environmental features.

18. The method of claim 17, further comprising:

displaying the 3D feature map of the environment in which the object is located on a display.

19. The method of claim 17, further comprising:

outputting the 3D feature map of the environment in which the object is located to a vehicle driver assistance system or autonomous vehicle system.

20. The method of claim 14, further comprising:

calculating a maximum likelihood estimation in accordance with a noise distribution model for each environmental feature in each image in which the environmental feature appears.

21. The method of claim 20, further comprising:

outputting the maximum likelihood estimations for each feature to a vehicle driver assistance system or autonomous vehicle system.

22. The method of claim 14, wherein, when selecting the group of images from the captured images, one captured image for each camera in the plurality of cameras in accordance with a time stamp of each image such that images in the group of images were captured at a common time.

23. A non-transitory machine readable medium having tangibly stored thereon executable instructions for execution by at least one processor of a computer vision system, the computer vision system comprising a processor, a memory coupled to the processor, and a plurality of cameras coupled to the processor, wherein the cameras are located about a host object, the plurality of cameras comprising at least two front facing cameras having a front facing view of an environment in which the object is located, at least one left side facing camera having a left side facing view of the environment in which the object is located, at least one right side facing camera having a right side facing view of the environment in which the object is located, and at least one rear facing camera having a rear facing view of the environment in which the object is located, wherein the executable instructions, when executed by the at least one processor, cause the computer vision system to:

select a group of images from images captured by the plurality of cameras, the group of images including one captured image for each camera in the plurality of cameras, which were captured at a common time;

perform image distortion compensation on each image in the group of images based on intrinsic parameters of the respective camera that captured the image;

rectify each image in the group of images so that all images in the group of images have a common image plane;

determine correspondence information for a plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on a relative position and alignment of each camera in the plurality of cameras;

determine depth information for each environmental feature in the plurality of environmental features for each image in the group of images in which a respective environmental feature is present based on the correspondence information of the environmental features; and

determine group optimized depth information for each environmental feature in the plurality of environmental features using the determined depth information of each respective environmental feature.