Three-dimensional model generation

• US 10,373,366 B2
• Filed: 09/17/2015
• Issued: 08/06/2019
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Bijan, Sarkis, Michel Adib, Sundaresan, Sairam
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Shin, Andrew
• Application Number:
  US14/857,620
• Publication Number:
  US 20160335792A1
• Time in Patent Office: 1,419 Days
• Field of Search: 345419, 345420, 345421, 345581, 345582, 345589, 345600, 345619, 382103, 382108, 382162, 382181, 382276
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/579 from motion

Abstract

A method for determining a pixel value of a texture pixel associated with a three-dimensional scan of an object includes prioritizing a sequence of image frames in a queue based on one or more prioritization parameters. The method also includes selecting a first image frame from the queue. The method also includes determining a pixel value of the particular texture pixel in the first image frame. The method further includes selecting a second image frame from the queue. The second image frame has a higher priority than the first image frame based on the one or more prioritization parameters. The method also includes modifying the pixel value of the particular texture pixel based on a pixel value of the particular texture pixel in the second image frame to generate a modified pixel value of the particular texture pixel.

Claims

1. An apparatus comprising:

interface circuitry configured to receive a sequence of image frames of an object, the sequence of image frames associated with a three-dimensional (3D) scan of the object; and

a processor configured to;

select, after receiving the sequence of images, a first image frame from the sequence of image frames, the first image frame having a first quality metric value, the first image frame depicting a particular texture pixel from a first angle;

determine a first pixel value of the particular texture pixel in the first image frame;

select a second image frame from the sequence of image frames, the second image frame having a second quality metric value that is higher than the first quality metric value, the second image frame depicting the particular texture pixel from a second angle;

modify the first pixel value based on a second pixel value of the particular texture pixel in the second image frame to generate a first modified pixel value of the particular texture pixel;

select a third image frame from the sequence of image frames, the third image frame having a third quality metric value that is higher than the second quality metric value and higher than the first quality metric value, the third image frame depicting the particular texture pixel from a third angle; and

modify the first modified pixel value based on a third pixel value of the particular texture pixel in the third image frame to generate a second modified pixel value of the particular texture pixel.

2. The apparatus of claim 1, wherein the second image frame has a higher priority than the first image frame based on one or more prioritization parameters, the one or more prioritization parameters based on the first quality metric value, and wherein the processor is further configured to generate a 3D model of the object.

3. The apparatus of claim 2, wherein the particular texture pixel in the 3D model has the second modified pixel value, and wherein the one or more prioritization parameters include an image frame timestamp weight, an image frame blur weight, an image frame color shift weight, an image frame face weight, an image frame distance weight, or image frame validity weight.

4. The apparatus of claim 3, wherein the first image frame has at least one of:

a higher degree of blurriness than the second image frame based upon the image frame blur weight;

a greater application of color correction than the second image frame based on the image frame color shift weight;

or

a greater capture distance from the object than the second image frame based on the image frame distance weight, and

wherein the one or more prioritization parameters are based on at least one of the image frame blur weight, the image frame color shift weight, or the image frame distance weight.

5. The apparatus of claim 1, wherein the second image frame has a higher priority than the first image frame based on the first quality metric value, and wherein the processor is further configured to:

after modifying the first modified pixel value, select another image frame from the sequence of image frames, wherein the other image frame has a higher quality metric value than the third quality metric value; and

modify the second modified pixel value based on a pixel value of the particular texture pixel in the other image frame.

6. The apparatus of claim 1, wherein the processor and the interface circuitry are integrated into a mobile device.

7. The apparatus of claim 1, wherein modifying the first pixel value comprises replacing the first pixel value with the second pixel value.

8. The apparatus of claim 1, wherein modifying the first pixel value comprises averaging the first pixel value with the second pixel value.

9. The apparatus of claim 1, wherein the first image frame is captured prior to the second image frame, and wherein the processor is further configured to assign, based on the first quality metric value, a higher priority, corresponding to a lower ordinal position within the sequence of image frames, to image frames depicting a larger portion of a detected face from a frontal perspective than to image frames depicting a smaller portion of the detected face, image frames depicting the larger portion of the detected face from the frontal perspective than to image frames depicting the larger portion of the detected face from perspectives other than the frontal perspective, or a combination thereof.

10. The apparatus of claim 1, wherein the processor is further configured to:

iteratively determine whether there is another image frame from the sequence of image frames in the sequence of image frames;

in response to a determination that there is another image frame in the sequence of image frames, modify the particular texture pixel based on a pixel value of the particular texture pixel in the other image frame; and

in response to a determination that each image frame of the sequence of image frames has been selected, generate a 3D model of the object, wherein the particular texture pixel in the 3D model has a modified pixel value that is based at least partially on a last selected image frame in the sequence of image frames.

11. The apparatus of claim 1, wherein the processor is further configured to prioritize image frames within the sequence of image frames based on a weighted quality metric, the weighted quality metric based on a quality metric associated with each image frame, the first image frame having a first prioritization value and the second image frame having a second prioritization value, the first prioritization value indicating that a first weight of the quality metric is less than a second weight of the quality metric, and wherein the first weight is less relevant to generation of a 3D model of the object than the second weight.

12. A method for determining a pixel value of a texture pixel associated with a three-dimensional (3D ) scan of an object, the method comprising:

selecting a first image frame from a sequence of image frames after receiving the sequence of images, the first image frame having a first quality metric value, the first image frame depicting a particular texture pixel from a first angle;

determining a first pixel value of the particular texture pixel in the first image frame;

selecting a second image frame from the sequence of image frames, the second image frame having a second quality metric value that is higher than the first quality metric value, the second image frame depicting the particular texture pixel from a second angle;

modifying the first pixel value based on a second pixel value of the particular texture pixel in the second image frame to generate a first modified pixel value of the particular texture pixel;

selecting a third image frame from the sequence of image frames, the third image frame having a third quality metric value that is higher than the second quality metric value and higher than the first quality metric value, the third image frame depicting the particular texture pixel from a third angle; and

modifying the first modified pixel value based on a third pixel value of the particular texture pixel in the third image frame to generate a second modified pixel value of the particular texture pixel.

13. The method of claim 12, wherein the second image frame has a higher priority than the first image frame based on one or more prioritization parameters, the one or more prioritization parameters based on a quality metric, and further comprising

determining a first prioritization value for the first image frame based on at least one of an identified first timestamp, a determined first frame face value, a determined first frame blur amount value, a determined first frame color correction amount value, a determined first frame distance value, an image frame timestamp weight, an image frame face weight, an image frame blur weight, an image frame color shift weight, or an image frame distance weight;

and

determining whether the first image frame is valid based on the first prioritization value, wherein a quality metric is based on the identified first timestamp, the determined first frame face value, the determined first frame blur amount value, the determined first frame color correction amount value, or the determined first frame distance value.

14. The method of claim 13, further comprising

determining a second prioritization value for the second image frame based on at least one of an identified second timestamp, a determined second frame face value, a determined second frame blur amount value, a determined second frame color correction amount value, a determined second frame distance value, the image frame timestamp weight, the image frame face weight, the image frame blur weight, the image frame color shift weight, or the image frame distance weight, wherein the sequence of image frames are prioritized by arranging the first image frame and the second image frame in the sequence of image frames based on the first prioritization value for the first image frame and the second prioritization value for the second image frame,

wherein the quality metric is based on the identified second timestamp, the determined second frame face value, the determined second frame blur amount value, the determined second frame color correction amount value, or the determined second frame distance value.

15. The method of claim 13, wherein determining whether the first image frame is valid comprises comparing the first prioritization value to a range of values, wherein the first image frame is determined to be valid based on the first prioritization value being within the range of values.

16. The method of claim 12, wherein the second image frame has a higher priority than the first image frame based on the first quality metric value, and further comprising:

after modifying the first modified pixel value, selecting another image frame from the sequence of image frames, wherein the other image frame has a higher quality metric value than the third quality metric value; and

modifying the second modified pixel value based on a pixel value of the particular texture pixel in the other image frame.

17. The method of claim 12, wherein the second image frame has a higher priority than the first image frame based on one or more prioritization parameters, the one or more prioritization parameters including at least one of an image frame timestamp weight, an image frame blur weight, an image frame color shift weight, an image frame face weight, an image frame distance weight, or an image frame validity weight, and wherein a higher priority corresponds to a lower ordinal position within the sequence of image frames.

18. The method of claim 17, wherein the first image frame is captured prior to the second image frame, and wherein a first value of the image frame timestamp weight applied to the first image frame is greater than a second value of the image frame timestamp weight applied to the second image frame based on first and second quality values assigned to the first image frame and the second image frame, respectively.

19. The method of claim 17, wherein the first image frame is captured prior to the second image frame, wherein a first order-of-processing priority value of the image frame timestamp weight applied to the first image frame is less than a second order-of-processing priority value of the image frame timestamp weight applied to the second image frame.

20. The method of claim 12, further comprising:

generating a 3D model of the object, the generating including the modifying of the first pixel value after the selecting of the first image frame and before the selecting of the third image frame;

generating, based on the sequence of image frames, a composite confidence map, the composite confidence map including pixel values for scanned pixels in the sequence of image frames and corresponding to the 3D model of the object; and

identifying one or more holes in the 3D model of the object based on the composite confidence map.

21. The method of claim 12, wherein the second image frame is determined to have a higher priority than the first image frame based on one or more prioritization parameters, the one or more prioritization parameters determined based on a quality metric, and further comprising:

determining a first frame face value of the first image frame;

determining a first prioritization value for the first image frame based on the first frame face value, an image frame face weight, or both; and

determining whether the first image frame is valid based on the first prioritization value, wherein the quality metric is based on the first frame face value.

22. A non-transitory computer-readable medium comprising instructions for determining a pixel value of a texture pixel associated with a three-dimensional (3D ) scan of an object, the instructions, when executed by a processor, cause the processor to perform operations comprising:

selecting a first image frame from a sequence of image frames after receiving the sequence of images, the first image frame having a first quality metric value, the first image frame depicting a particular texture pixel from a first angle;

determining a first pixel value of the particular texture pixel in the first image frame;

selecting a second image frame from the sequence of image frames, the second image frame having a second quality metric value that is higher than the first quality metric value, the second image frame depicting the particular texture pixel from a second angle;

modifying the first pixel value based on a second pixel value of the particular texture pixel in the second image frame to generate a first modified pixel value of the particular texture pixel;

selecting a third image frame from the sequence of image frames, the third image frame having a third quality metric value that is higher than the second quality metric value and higher than the first quality metric value, the third image frame depicting the particular texture pixel from a third angle; and

modifying the first modified pixel value based on a third pixel value of the particular texture pixel in the third image frame to generate a second modified pixel value of the particular texture pixel.

23. The non-transitory computer-readable medium of claim 22, the operations further comprising determining one or more prioritization parameters for each of the first image frame and the second image frame, wherein the second image frame has a higher priority than the first image frame and the third image frame has a higher priority than the second image frame.

24. The non-transitory computer-readable medium of claim 23, wherein the one or more prioritization parameters comprise one or more of an image frame timestamp weight, an image frame blur weight, an image frame color shift weight, an image frame face weight, a distance weight, or an image frame validity weight, wherein the operations further comprise generating a 3D model of the object, and wherein the particular texture pixel in the 3D model has the second modified pixel value.

25. The non-transitory computer-readable medium of claim 23, wherein a higher priority of a selected image frame corresponds to a lower ordinal position of the selected image frame within the sequence of image frames.

26. The non-transitory computer-readable medium of claim 22, wherein the second image frame has a higher priority than the first image frame based on one or more prioritization parameters, the one or more prioritization parameters based on the first quality metric value, and wherein the operations further comprise:

determining, based on the one or more prioritization parameters, a prioritization value for a third image frame associated with the sequence of image frames; and

determining whether the third image frame is valid based on the prioritization value.

27. The non-transitory computer-readable medium of claim 26, wherein the operations further comprise discarding the third image frame based on a determination that the third image frame is invalid.

28. The non-transitory computer-readable medium of claim 22, further comprising prioritizing another image frame in a priority buffer to be processed and modified before the first image frame, wherein the first image frame is captured prior to the second image frame, and wherein a first value of an image frame timestamp weight applied to the first image frame is greater than a second value of the image frame timestamp weight applied to the second image frame.

29. The non-transitory computer-readable medium of claim 22, wherein the first image frame is captured prior to the second image frame, wherein one or more prioritization parameters include an image frame timestamp weight, and wherein the first image frame is selected for processing and modification based on a first value of the image frame timestamp weight applied to the first image frame being less than a second value of the image frame timestamp weight applied to the second image frame.

30. An apparatus comprising:

means for selecting, after receiving the sequence of images, a first image frame from a sequence of image frames, the sequence of image frames associated with a three-dimensional (3D ) scan of an object, the first image frame having a first quality metric value, the first image frame depicting a particular texture pixel from a first angle;

means for determining a first pixel value of the particular texture pixel in the first image frame;

means for selecting a second image frame from the sequence of image frames, the second image frame having a second quality metric value that is higher than the first quality metric value, the second image frame depicting the particular texture pixel from a second angle;

means for modifying the first pixel value based on a second pixel value of the particular texture pixel in the second image frame to generate a first modified pixel value of the particular texture pixel;

means for selecting a third image frame from the sequence of image frames, the third image frame having a third quality metric value that is higher than the second quality metric value and higher than the first quality metric value, the third image frame depicting the particular texture pixel from a third angle; and

means for modifying the first modified pixel value based on a third pixel value of the particular texture pixel in the third image frame to generate a second modified pixel value of the particular texture pixel.

31. The apparatus of claim 30, further comprising means for generating a 3D model of the object, wherein the particular texture pixel in the 3D model has the second modified pixel value.

32. The apparatus of claim 30, wherein the second image frame has a higher priority than the first image frame based on one or more prioritization parameters that are based on the first quality metric value, and wherein the one or more prioritization parameters include the one or more of an image frame timestamp weight, an image frame blur weight, an image frame color shift weight, an image frame face weight, an image frame distance weight, or an image frame validity weight.