Depth sensor noise

• US 10,586,309 B2
• Filed: 12/19/2017
• Issued: 03/10/2020
• Est. Priority Date: 12/28/2016
• Status: Active Grant

• Current Assignee: Dassault Systemes SA
• Original Assignee: Dassault Systemes SA
• Inventors: Ayari, Mohamed Amine, Guitteny, Vincent
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/846,328
• Publication Number:
  US 20180182071A1
• Time in Patent Office: 812 Days
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

Described is a computer-implemented method for learning a function configured for reconstructing, for a class of real objects, a 3D modeled object that represents an instance of the class from a depth map of the instance. The method comprises providing a parametric model of the class, creating a database, and learning the function with the database. The method improves the 3D reconstruction of a real object from a depth map.

Claims

1. A computer-implemented method for determining a function configured for adding a noise of a type of depth sensor, the type of depth sensor being a category of depth sensors of a similar or an at least substantially same constitution such that depth sensors present a similar or an at least substantially same noise, to an input depth map, the method comprising:

forming a learning dataset, the learning dataset including a plurality of noiseless depth maps each associated to a respective noisy depth map, each noiseless depth map and respective noisy depth map corresponding to a same one of a plurality of depth sensor positionings in one or more scenes, the forming including;

acquiring each noisy depth map by performing depth measurements with a physical instance of the type of depth sensor, the physical instance of the type of depth sensor being performed with a depth sensor of the type of depth sensor, and

virtually calculating each noiseless depth map by determining each noiseless depth map fully based on numerical data and without any physical acquisition; and

learning the function based on the learning dataset, the learning of the function including adjusting parameters of the function by performing machine-learning on the learning dataset such that applying the learnt function to a given noiseless depth map of the learning dataset leads to a result corresponding to the respective noisy depth map associated with the given noiseless depth map in the dataset.

2. The method of claim 1, wherein the virtually calculating each noiseless depth map is based on the corresponding depth sensor positioning and on a predetermined model of the corresponding scene.

3. The method of claim 1, wherein the forming further includes determining the corresponding depth sensor positioning from the respective noisy depth map.

4. The method of claim 3, wherein determining the corresponding depth sensor positioning from the respective noisy depth map includes identifying in the respective noisy depth map a calibration object and/or a calibration pattern.

5. The method of claim 1, wherein the plurality of depth sensor positionings includes one or more sub-pluralities of depth sensor positionings that each correspond to a respective depth from a same scene region.

6. The method of claim 5, wherein each sub-plurality of depth sensor positionings, that each corresponds to a respective depth from a same scene region, includes more than 10 depth sensor positionings.

7. The method of claim 5, wherein each sub-plurality of depth sensor positionings, that each corresponds to a respective depth from a same scene region, includes more than 50 depth sensor positionings.

8. The method of claim 5, wherein the plurality of noisy depth maps forms one or more videos.

9. The method of claim 1, wherein learning the function is performed by regression learning.

10. The method of claim 1, wherein learning the function is performed within a space of networks comprising convolutional layers and/or deconvolutional layers.

11. The method of claim 10, wherein the convolutional layers and/or deconvolutional layers implements filters each of a size inferior to 10×

10.

12. The method of claim 10, wherein the convolutional layers and/or deconvolutional layers implements filters each of a size inferior to 5×

5.

13. The method of claim 10, wherein the convolutional layers and/or deconvolutional layers implements filters each of a size equal to 3×

3.

14. A non-transitory computer readable medium having recorded thereon a function learnable according to a computer-implemented method for determining a function configured for adding a noise of a type of depth sensor, the type of depth sensor being a category of depth sensors of a similar or an at least substantially same constitution such that depth sensors present a similar or an at least substantially same noise, to an input depth map, the method comprising:

forming a learning dataset, the learning dataset including a plurality of noiseless depth maps each associated to a respective noisy depth map, each noiseless depth map and respective noisy depth map corresponding to a same one of a plurality of depth sensor positionings in one or more scenes, the forming including;

acquiring each noisy depth map by performing depth measurements with a physical instance of the type of depth sensor, the physical instance of the type of depth sensor being performed with a depth sensor of the type of depth sensor, and

virtually calculating each noiseless depth map by determining each noiseless depth map fully based on numerical data and without any physical acquisition; and

learning the function based on the learning dataset, the learning of the function including adjusting parameters of the function by performing machine-learning on the learning dataset such that applying the learnt function to a given noiseless depth map of the learning dataset leads to a result corresponding to the respective noisy depth map associated with the given noiseless depth map in the dataset.

15. A computer-implemented method for generating a depth map based on a depth sensor positioning in a scene and on a predetermined model of the scene, the method comprising:

virtually calculating a noiseless depth map based on the depth sensor positioning and on the predetermined model of the scene; and

applying, to the calculated noiseless depth map, a function learnable according to a further computer-implemented method for determining a function configured for adding a noise of a type of depth sensor, the type of depth sensor being a category of depth sensors of a similar or an at least substantially same constitution such that depth sensors present a similar or an at least substantially same noise, to an input depth map, including;

forming a learning dataset, the learning dataset including a plurality of noiseless depth maps each associated to a respective noisy depth map, each noiseless depth map and respective noisy depth map corresponding to a same one of a plurality of depth sensor positionings in one or more scenes, the forming including;

acquiring each noisy depth map by performing depth measurements with a physical instance of the type of depth sensor, the physical instance of the type of depth sensor being performed with a depth sensor of the type of depth sensor, and

virtually calculating each noiseless depth map by determining each noiseless depth map fully based on numerical data and without any physical acquisition; and

learning the function based on the learning dataset, the learning of the function including adjusting parameters of the function by performing machine-learning on the learning dataset such that applying the learnt function to a given noiseless depth map of the learning dataset leads to a result corresponding to the respective noisy depth map associated with the given noiseless depth map in the dataset.

16. A non-transitory computer readable medium having recorded thereon a computer program comprising instructions for performing a computer-implemented method for determining a function configured for adding a noise of a type of depth sensor, the type of depth sensor being a category of depth sensors of a similar or an at least substantially same constitution such that depth sensors present a similar or an at least substantially same noise, to an input depth map, the method comprising:

forming a learning dataset, the learning dataset including a plurality of noiseless depth maps each associated to a respective noisy depth map, each noiseless depth map and respective noisy depth map corresponding to a same one of a plurality of depth sensor positionings in one or more scenes, the forming including;

acquiring each noisy depth map by performing depth measurements with a physical instance of the type of depth sensor, the physical instance of the type of depth sensor being performed with a depth sensor of the type of depth sensor, and

virtually calculating each noiseless depth map by determining each noiseless depth map fully based on numerical data and without any physical acquisition; and

learning the function based on the learning dataset, the learning of the function including adjusting parameters of the function by performing machine-learning on the learning dataset such that applying the learnt function to a given noiseless depth map of the learning dataset leads to a result corresponding to the respective noisy depth map associated with the given noiseless depth map in the dataset.

17. A computer system comprising:

a processor coupled to the memory, the memory having recorded thereon a computer program comprising instructions for performing a computer-implemented method for determining a function configured for adding a noise of a type of depth sensor, the type of depth sensor being a category of depth sensors of a similar or an at least substantially same constitution such that depth sensors present a similar or an at least substantially same noise, to an input depth map, the instructions when executed by the processor cause the process to be configured to;

form a learning dataset, the learning dataset including a plurality of noiseless depth maps each associated to a respective noisy depth map, each noiseless depth map and respective noisy depth map corresponding to a same one of a plurality of depth sensor positionings in one or more scenes, the processor being configured to form the learning dataset by being further configured to

acquire each noisy depth map by performing depth measurements with a physical instance of the type of depth sensor, the physical instance of the type of depth sensor being performed with a depth sensor of the type of depth sensor, and

virtually calculate each noiseless depth map by determining each noiseless depth map fully based on numerical data and without any physical acquisition; and

learn the function based on the learning dataset, the learning of the function including adjusting parameters of the function by performing machine-learning on the learning dataset such that applying the learnt function to a given noiseless depth map of the learning dataset leads to a result corresponding to the respective noisy depth map associated with the given noiseless depth map in the dataset.

18. The system of claim 17, wherein the processor is further configured to virtually calculate each noiseless depth map based on the corresponding depth sensor positioning and on a predetermined model of the corresponding scene.

19. The system of claim 17, wherein the processor is further configured to form the learning data set by being further configured to determine the corresponding depth sensor positioning from the respective noisy depth map.

20. The system of claim 19, wherein the processor is further configured to determine the corresponding depth sensor positioning from the respective noisy depth map by being further configured to identify in the respective noisy depth map a calibration object and/or a calibration pattern.

21. The system of claim 17, wherein the plurality of depth sensor positionings includes one or more sub-pluralities of depth sensor positionings that each correspond to a respective depth from a same scene region.

22. The system of claim 21, wherein each sub-plurality of depth sensor positionings, that each corresponds to a respective depth from a same scene region, includes more than 10 depth sensor positionings.

23. The system of claim 21, wherein each sub-plurality of depth sensor positionings, that each corresponds to a respective depth from a same scene region, includes more than 50 depth sensor positionings.

24. The system of claim 21, wherein the plurality of noisy depth maps forms one or more videos.