Electronic system including image processing unit for reconstructing 3D surfaces and iterative triangulation method

• US 10,462,447 B1
• Filed: 06/28/2019
• Issued: 10/29/2019
• Est. Priority Date: 12/14/2015
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kohle, Roderick, Michielin, Francesco, Harres, Dennis
• Primary Examiner(s): Ahmed, Samir A
• Application Number:
  US16/456,612
• Publication Number:
  US 20190320154A1
• Time in Patent Office: 123 Days
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An electronic system includes a circuitry configured to obtain a sequence of frames of an object under different viewing angles at consecutive time instances. For a first time instance, the circuitry generates a point cloud descriptive for an external surface of the object on basis of (i) a point cloud obtained for a second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance.

Claims

1. An electronic system, comprising

circuitry configured to

obtain a sequence of frames of an object under different viewing angles,

process the sequence of frames, wherein the circuitry for processing the sequence of frames is further configured to

receive preconditioned frames,

for each preconditioned frame, extract salient image points, wherein the salient image points identify small image areas of high contrast,

reduce a total number of salient image points by selecting only a predetermined number of stable salient image points, and

generate, for a first time instance, a point cloud descriptive for an external surface of the object based on (i) a point cloud obtained for a second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance,

display a 3D representation of the point cloud generated for the first time instance, and

update the displayed 3D representation while obtaining the sequence of frames.

2. The electronic system according to claim 1, wherein

the disparity information concerns the frame captured at the first time instance and a frame captured at the second time instance.

3. The electronic system according to claim 1, wherein

the frame captured at the first time instance includes one or more sub-frames captured at the first time instance from different positions.

4. The electronic system according to claim 1, wherein

the circuitry is further configured to identify sparse correspondences between the first frame and a preceding frame for sparse salient image points in the first and the preceding frame, the preceding frame corresponding to a frame captured at the second time instance.

5. The electronic system according to claim 4, wherein

the circuitry is further configured to calculate dense correspondences between two consecutive frames, a starting value being obtained by the preceding point cloud updated by information obtained about the identified sparse correspondences.

6. The electronic system according to claim 5, wherein

the circuitry is further configured to calculate an actual pose of an image capturing unit relative to the object based on the dense correspondences dc(n) for the first frame fr(n) by using iterative methods.

7. The electronic system according to claim 1, wherein

the circuitry is further configured to update depth information contained in the point cloud obtained at the second time instance by using information contained in the first frame.

8. The electronic system according to claim 7, wherein

the circuitry uses a ray accumulator configured to temporarily store a ten coefficient vector.

9. The electronic system according to claim 1, wherein the circuitry comprises,

an image capturing unit adapted to obtain the sequence of frames of the object under different viewing angles;

and

a processing unit adapted to generate, for the first time instance, the point cloud descriptive for the external surface of the object based on (i) the point cloud obtained for the second time instance preceding the first time instance and (ii) disparity information concerning the frame captured at the first time instance.

10. A 3D reconstruction method, comprising

capturing a sequence of frames of an object under different viewing angles at consecutive time instances;

processing the sequence of frames, wherein processing the sequence of frames includes

receiving preconditioned frames;

for each preconditioned frame, extracting salient image points, wherein the salient image points identify small image areas of high contrast;

reducing a total number of salient image points by selecting only a predetermined number of stable salient image points; and

generating, for a first time instance, a point cloud descriptive for an external surface of the object based on (i) a point cloud obtained for second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance;

displaying a 3D representation of the point cloud generated for the first time instance; and

updating the displayed 3D representation while capturing the sequence of frames.

11. The 3D reconstruction method according to claim 10, wherein

the disparity information concerns the frame captured at the first time instance and a frame captured at the second time instance.

12. The 3D reconstruction method according to claim 10, wherein

the frame captured at the first time instance includes one or more sub-frames captured at the first time instance from different positions.