Three-dimensional model generation

• US 9,911,242 B2
• Filed: 09/17/2015
• Issued: 03/06/2018
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sundaresan, Sairam, Sarkis, Michel Adib, Forutanpour, Bijan, Qi, Yingyong
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US14/856,990
• Publication Number:
  US 20160335782A1
• Time in Patent Office: 901 Days
• Field of Search: None
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 1/56 Processing of colour picture signals H04N1/52 takes precedence
• CPC Class Codes: H04N 1/60 Colour correction or control H04N1/54 takes precedence

Abstract

A method for adjusting pixel colors between image frames includes scanning, at a processor, a first image frame of a sequence of image frames. The method also includes determining a grayscale threshold based on characteristics of the first image frame to identify gray pixel candidates in the first image frame. The method further includes adjusting a pixel value of each pixel in the first image frame based on a chromatic adaptation transform estimation. The chromatic adaptation transform estimation is based on the gray pixel candidates. The grayscale threshold may be computed for each image frame in the sequence of image frames.

Claims

1. An apparatus comprising:

interface circuitry configured to receive image frames of an object, the image frames associated with a three-dimensional scan of the object; and

a processor configured to;

determine a grayscale threshold based on characteristics of a first image frame of a sequence of the image frames;

identify gray pixel candidates in the first image frame based on the grayscale threshold;

determine a mean pixel value of the gray pixel candidates; and

adjust a pixel value of each pixel in the first image frame based on a chromatic adaptation transform estimation, the chromatic adaptation transform estimation based on the mean pixel value.

2. The apparatus of claim 1, wherein the processor is further configured to:

compare a first chrominance component value and a second chrominance component value for each pixel in the first image frame to the grayscale threshold; and

identify a particular pixel in the first image frame as a gray pixel candidate if a first chrominance component value for the particular pixel and a second chrominance component value for the particular pixel satisfy the grayscale threshold.

3. The apparatus of claim 1, wherein the processor is further configured to:

determine the mean pixel value based on a first chrominance component value and a second chrominance component value of each of the gray pixel candidates; and

estimate a chromatic adaptation transform between the mean pixel value and a pixel value corresponding to a reference white light.

4. The apparatus of claim 1, wherein the processor is further configured to:

determine a difference between the chromatic adaptation transform estimation and a preceding chromatic adaptation transform estimation; and

compare the difference to a convergence threshold.

5. The apparatus of claim 4, wherein the pixel value is adjusted based on the chromatic adaptation transform estimation if the difference fails to satisfy the convergence threshold.

6. The apparatus of claim 4, wherein the processor is further configured to:

determine a subsequent grayscale threshold based on characteristics of the first image frame if the difference satisfies the convergence threshold, the subsequent grayscale threshold used to identify gray pixel candidates in the first image frame; and

adjust a pixel value of each pixel in the first image frame based on a subsequent chromatic adaptation transform estimation.

7. The apparatus of claim 1, wherein the processor is further configured to:

determine a color cast of the first image frame based on first chrominance component values for each off-gray point in the first image frame and second chrominance component values for each off-gray point in the first image frame; and

determine the grayscale threshold based on the color cast of the first image frame, wherein the grayscale threshold at least partially offsets the color cast of the first image frame.

8. The apparatus of claim 1, wherein the processor is further configured to:

determine a color cast of the first image frame based on a histogram indicating a proportion of off-gray points having a yellow-based tint to off-gray points having a blue-based tint; and

determine the grayscale threshold based on the color cast of the first image frame.

9. The apparatus of claim 1, wherein each adjusted pixel value is used in a three-dimensional model of the object.

10. The apparatus of claim 1, wherein the processor and the interface circuitry are integrated into a mobile device, an aircraft, an automobile, a drone, a robot, a personal camera, a mounted camera, an unmanned vehicle, or a processing system communicatively coupled to one or more mounted cameras.

11. A method for adjusting pixel colors between image frames, the image frames associated with a three-dimensional scan of an object, the method comprising:

determining, at a processor, a grayscale threshold based on characteristics of a first image frame of a sequence of image frames;

identifying gray pixel candidates in the first image frame based on the grayscale threshold;

determining a mean pixel value of the gray pixel candidates; and

adjusting a pixel value of each pixel in the first image frame based on a chromatic adaptation transform estimation, the chromatic adaptation transform estimation based on the mean pixel value.

12. The method of claim 11, wherein identifying the gray pixel candidates comprises:

comparing a first chrominance component value and a second chrominance component value for each pixel in the first image frame to the grayscale threshold; and

identifying a particular pixel in the first image frame as a gray pixel candidate if a first chrominance component value for the particular pixel and a second chrominance component value for the particular pixel satisfy the grayscale threshold.

13. The method of claim 11, further comprising:

determining the mean pixel value based on a first chrominance component value and a second chrominance component value of each of the gray pixel candidates; and

estimating a chromatic adaptation transform between the mean pixel value and a pixel value corresponding to a reference white light.

14. The method of claim 11, further comprising:

determining a difference between the chromatic adaptation transform estimation and a preceding chromatic adaptation transform estimation; and

comparing the difference to a convergence threshold.

15. The method of claim 14, wherein the pixel value is adjusted based on the chromatic adaptation transform estimation if the difference fails to satisfy the convergence threshold.

16. The method of claim 14, further comprising:

determining a subsequent grayscale threshold based on characteristics of the first image frame if the difference satisfies the convergence threshold, the subsequent grayscale threshold used to identify gray pixel candidates in the first image frame; and

adjusting a pixel value of each pixel in the first image frame based on a subsequent chromatic adaptation transform estimation.

17. The method of claim 11, further comprising:

determining a color cast of the first image frame based on first chrominance component values for each off-gray point in the first image frame and second chrominance component values for each off-gray point in the first image frame; and

determining the grayscale threshold based on the color cast of the first image frame, wherein the grayscale threshold at least partially offsets the color cast of the first image frame.

18. The method of claim 11, further comprising:

determining a color cast of the first image frame based on a histogram indicating a proportion of off-gray points having a yellow-based tint to off-gray points having a blue-based tint; and

determining the grayscale threshold based on the color cast of the first image frame.

19. The method of claim 11, wherein the adjusted pixel value is used in a three-dimensional model of the object.

20. The method of claim 11, wherein the processor is integrated into a mobile device, an aircraft, an automobile, a drone, a robot, a personal camera, a mounted camera, an unmanned vehicle, or a processing system communicatively coupled to one or more mounted cameras.

21. A non-transitory computer-readable medium comprising instructions for adjusting pixel colors between image frames, the image frames associated with a three-dimensional scan of an object, the instructions, when executed by a processor, cause the processor to perform operations comprising:

determining a grayscale threshold based on characteristics of a first image frame of a sequence of the image frames;

identifying gray pixel candidates in the first image frame based on the grayscale threshold;

determining a mean pixel value of the gray pixel candidates; and

adjusting a pixel value of each pixel in the first image frame based on a chromatic adaptation transform estimation, the chromatic adaptation transform estimation based on the mean pixel value.

22. The non-transitory computer-readable medium of claim 21, wherein identifying the gray pixel candidates comprises:

comparing a first chrominance component value and a second chrominance component value for each pixel in the first image frame to the grayscale threshold; and

identifying a particular pixel in the first image frame as a gray pixel candidate if a first chrominance component value for the particular pixel and a second chrominance component value for the particular pixel satisfy the grayscale threshold.

23. The non-transitory computer-readable medium of claim 21, wherein the operations further comprise:

determining the mean pixel value based on a first chrominance component value and a second chrominance component value of each of the gray pixel candidates; and

estimating a chromatic adaptation transform between the mean pixel value and a pixel value corresponding to a reference white light.

24. The non-transitory computer-readable medium of claim 21, wherein the operations further comprise:

determining a difference between the chromatic adaptation transform estimation and a preceding chromatic adaptation transform estimation; and

comparing the difference to a convergence threshold.

25. The non-transitory computer-readable medium of claim 24, wherein the pixel value is adjusted if the difference fails to satisfy the convergence threshold.

26. An apparatus comprising:

means for determining a grayscale threshold based on characteristics of a first image frame of a sequence of image frames;

means for identifying gray pixel candidates in the first image frame using a processor, the gray pixel candidates identified based on the grayscale threshold;

means for determining a mean pixel value of the gray pixel candidates; and

means for adjusting a pixel value of each pixel in the first image frame based on a chromatic adaptation transform estimation, the chromatic adaptation transform estimation based on the mean pixel value.