HYBRID DEPTH SENSING PIPELINE

• US 20150049169A1
• Filed: 08/15/2013
• Published: 02/19/2015
• Est. Priority Date: 08/15/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Krig, Scott, Johnson, Gregory
• Granted Patent: US 10,497,140 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format

Abstract

An apparatus for a hybrid tracking and mapping is described herein. The apparatus includes logic to determine a plurality of depth sensing techniques. The apparatus also includes logic to vary the plurality of depth sensing techniques based on a camera configuration. Additionally, the apparatus includes logic to generate a hybrid tracking and mapping pipeline based on the depth sensing techniques and the camera configuration.

Claims

1. An apparatus for hybrid tracking and mapping, comprising:

logic to determine a plurality of depth sensing techniques;

logic to vary the plurality of depth sensing techniques based on a sensor configuration;

logic to vary the plurality of depth sensing techniques based on a compute unit configuration; and

logic to generate a hybrid tracking and mapping pipeline based on the depth sensing techniques, sensor configuration, and the compute unit configuration.

2. The apparatus of claim 1, wherein the depth sensing techniques include disparity estimation from stereoscopic image left/right pairs, virtual stereoscopic frames, multi-view stereoscopic (MVS) reconstruction from image sequences, and structured light scanning.

3. The apparatus of claim 1, wherein the compute unit configuration includes a location of the unit, the movement of the unit, and the power available to the unit.

4. The apparatus of claim 1, wherein the pipeline is regenerated based on a new compute unit configuration.

5. The apparatus of claim 1, wherein the pipeline is optimized based on the compute unit configuration.

6. The apparatus of claim 1, wherein algorithm tuning is used to generate the hybrid tracking and mapping pipeline.

7. The apparatus of claim 1, wherein the hybrid tracking and mapping pipeline is used to produce a variable resolution depth representation.

8. The apparatus of claim 1, wherein the apparatus is a tablet device, print device, or smart phone.

9. An image capture device including at least one sensor, wherein the sensor captures depth information that is processed by a hybrid tracking and mapping pipeline, wherein the hybrid tracking and mapping pipeline includes a plurality of depth sensing techniques.

10. The image capture device of claim 9, wherein the depth sensing techniques include disparity estimation from stereoscopic image left/right pairs, virtual stereoscopic frames, multi-view stereoscopic (MVS) reconstruction from image sequences, and structured light scanning.

11. The image capture device of claim 9, wherein the plurality of depth sensing techniques varies based on a camera configuration.

12. The image capture device of claim 9, wherein virtual stereoscopic reconstruction is performed by rectifying, aligning, and cropping sequential image data.

13. The image capture device of claim 9, wherein neighboring pixels or features are used to interpolate values within a stereoscopic left/right frame pair.

14. The image capture device of claim 9, wherein a bundle of past frames, a past point cloud, an image pyramid buffer, or any combination thereof are used in an MVS reconstruction stage to improve the depth map.

15. The image capture device of claim 9, wherein the hybrid tracking and mapping pipeline generates a depth representation.

16. A system for hybrid tracking and mapping, comprising:

a display;

a radio;

a memory that is to store instructions and that is communicatively coupled to the display; and

a processor communicatively coupled to the radio and the memory, wherein when the processor is to execute the instructions, the processor is to;

determine a plurality of depth sensing techniques;

vary the plurality of depth sensing techniques based on a camera configuration; and

generate a hybrid tracking and mapping pipeline based on the depth sensing techniques and the camera configuration.

17. The system of claim 16, wherein the depth sensing techniques include disparity estimation from stereoscopic image left/right pairs, virtual stereoscopic frames, multi-view stereoscopic (MVS) reconstruction from image sequences, and structured light scanning.

18. The system of claim 16 wherein the camera configuration includes a sensor type, an amount of power available, and a resolution of an image sensor.

19. The system of claim 16, wherein algorithm tuning is used to generate the hybrid tracking and mapping pipeline.

20. The system of claim 16, wherein the hybrid tracking and mapping pipeline is used to produce a variable resolution depth representation.

21. A method for hybrid tracking and mapping, comprising:

determining a plurality of depth sensing techniques;

varying the plurality of depth sensing techniques based on a camera configuration, wherein the camera configuration is a monocular camera that produces a sequence of frames; and

generating a hybrid tracking and mapping pipeline based on the depth sensing techniques and the camera configuration.

22. The method of claim 21, further comprising generating a virtual stereo pair of frames using a frame N and a frame N−

1 of the sequence of frames by;

locating fiducial marks in the sequence of frames;

determining the orientation of a camera pose; and

performing an affine transform on the frame N and the frame N−

1.

23. The method of claim 22, wherein the virtual stereo pair of frames is used to generate depth information.