Three-dimensional model generation

• US 10,304,203 B2
• Filed: 09/17/2015
• Issued: 05/28/2019
• Est. Priority Date: 05/14/2015
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Bijan, Sarkis, Michel Adib, Sundaresan, Sairam
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Shin, Andrew
• Application Number:
  US14/857,289
• Publication Number:
  US 20160335809A1
• Time in Patent Office: 1,349 Days
• Field of Search: 345419, 345420, 345427, 345581, 345582, 345589, 345619, 382108, 382276, 382312, 382313
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/579 from motion

Abstract

A method for texture reconstruction associated with a three-dimensional scan of an object includes scanning, at a processor, a sequence of image frames captured by an image capture device at different three-dimensional viewpoints. The method also includes generating a composite confidence map based on the sequence of image frames. The composite confidence map includes pixel values for scanned pixels in the sequence of image frames. The method further includes identifying one or more holes of a three-dimensional model based on the composite confidence map.

Claims

1. An apparatus comprising:

interface circuitry configured to receive a sequence of image frames associated with a three-dimensional scan of an object; and

a processor configured to;

iteratively generate a composite confidence map based on the sequence of image frames, the composite confidence map including pixel values for scanned pixels in the sequence of image frames;

identify, in the composite confidence map, a particular pixel value for a texture coordinate for a three-dimensional (3D) model, the 3D model generated based on the sequence of image frames;

identify a hole triangle based on the particular pixel value corresponding to a black value; and

composite texture over the hole triangle using the composite confidence map as an alpha channel.

2. The apparatus of claim 1, wherein the processor is further configured to fill the hole triangle using color-per-vertex rendering.

3. The apparatus of claim 1, wherein the processor and the interface circuitry are integrated into a mobile device.

4. The apparatus of claim 1, wherein the processor and the interface circuitry are integrated into an aircraft, an automobile, a drone, a robot, a camera, an unmanned vehicle, or a processing system communicatively coupled to one or more mounted cameras.

5. The apparatus of claim 1, wherein generating the composite confidence map includes updating pixel value weights for the scanned pixels based on a respective pixel value of each respective image frame and based on a camera pose associated with each respective image frame, the camera pose including a camera position and a camera orientation.

6. The apparatus of claim 1, wherein the particular triangle represents a hole border vertex if the particular pixel value is grey.

7. The apparatus of claim 1, wherein the apparatus is a wireless communication device, further comprising:

a transmitter configured to transmit the composite confidence map.

8. The apparatus of claim 7, wherein the wireless communication device is a cellular telephone and the composite confidence map is transmitted by the transmitter and modulated according to a cellular communication standard.

9. The apparatus of claim 1, wherein the processor is further configured to:

iteratively generate a camera pose error correction matte for each frame of the sequence of image frames, the camera pose error correction matte indicating a boundary of the object from a background;

iteratively generate a camera seam matte for each frame of the sequence of image frames; and

iteratively generate a color texture map for each frame of the sequence of image frames based on the camera pose error correction matte and the camera seam matte.

10. The apparatus of claim 1, wherein the processor is further configured to:

generate an updated 3D model based on the composited texture over the hole triangle.

11. A method for texture reconstruction associated with a three-dimensional scan of an object, the method comprising:

scanning, at a processor, a sequence of image frames captured by an image capture device, wherein each image frame in the sequence of image frames is captured at a different three-dimensional viewpoint;

iteratively generating a composite confidence map based on the sequence of image frames, the composite confidence map including pixel values for scanned pixels in the sequence of image frames;

identifying, in the composite confidence map, a particular pixel value for a texture coordinate for a three-dimensional (3D) model, the 3D model generated based on the sequence of image frames; and

identifying a hole triangle based on the particular pixel value corresponding to a black value; and

compositing texture over the hole triangle using the composite confidence map as an alpha channel.

12. The method of claim 11, further comprising filling the hole triangle using color-per-vertex rendering.

13. The method of claim 11, wherein the processor and the image capture device are integrated into a mobile device.

14. The method of claim 11, wherein the processor and the image capture device are integrated into an aircraft, an automobile, a drone, a robot, a camera, an unmanned vehicle, or a processing system communicatively coupled to one or more mounted cameras.

15. The method of claim 11, the method being executable on a wireless communication device, wherein the wireless device comprises:

a memory configured to store the sequence of image frames;

the processor configured to execute instructions to iteratively generate the composite confidence map based on the sequence of image frames stored in the memory; and

a transmitter configured to transmit the composite confidence map.

16. The method of claim 15, wherein the wireless communication device is a cellular telephone and the composite confidence map is transmitted by the transmitter and modulated according to a cellular communication standard.

17. The method of claim 11, wherein the method further comprises:

iteratively generating a camera pose error correction matte for each frame of the sequence of image frames, the camera pose error correction matte indicating a boundary of the object from a background;

iteratively generating a camera seam matte for each frame of the sequence of image frames; and

iteratively generating a color texture map for each frame of the sequence of image frames based on the camera pose error correction matte and the camera seam matte.

18. The method of claim 11, wherein the method further comprises:

generating an updated 3D model based on the composited texture over the hole triangle.

19. A non-transitory computer-readable medium comprising instructions for texture reconstruction associated with a three-dimensional scan of an object, the instructions, when executed by processor, cause the processor to perform operations comprising:

scanning a sequence of image frames captured by an image capture device, wherein each image frame in the sequence of image frames is captured at a different three-dimensional viewpoint;

iteratively generating a composite confidence map based on the sequence of image frames, the composite confidence map including pixel values for scanned pixels in the sequence of image frames;

identifying, in the composite confidence map, a particular pixel value for a texture coordinate for a three-dimensional (3D) model, the 3D model generated based on the sequence of image frames;

identifying a hole triangle based on the particular pixel value corresponding to a black value; and

compositing texture over the hole triangle using the composite confidence map as an alpha channel.

20. The non-transitory computer-readable medium of claim 19, wherein the operations further comprise filling the hole triangle using color-per-vertex rendering.

21. The non-transitory computer-readable medium of claim 19, wherein the processor and the image capture device are integrated into a mobile device.

22. The non-transitory computer-readable medium of claim 19, wherein the processor and the image capture device are integrated into an aircraft, an automobile, a drone, a robot, a camera, an unmanned vehicle, or a processing system communicatively coupled to one or more mounted cameras.

23. The non-transitory computer-readable medium of claim 19, wherein the operations further comprise:

iteratively generating a camera pose error correction matte for each frame of the sequence of image frames, the camera pose error correction matte indicating a boundary of the object from a background;

iteratively generating a camera seam matte for each frame of the sequence of image frames; and

iteratively generating a color texture map for each frame of the sequence of image frames based on the camera pose error correction matte and the camera seam matte.

24. The non-transitory computer-readable medium of claim 19, wherein the operations further comprise:

generating an updated 3D model based on the composited texture over the hole triangle.