Color extraction of a video stream

• US 10,339,410 B1
• Filed: 01/13/2016
• Issued: 07/02/2019
• Est. Priority Date: 01/13/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Gusarov, Maksim Igorevich
• Primary Examiner(s): Chen, Xuemei G
• Application Number:
  US14/994,946
• Time in Patent Office: 1,266 Days
• Field of Search: 345102, 348143, 358 19, 358518, 382166, 382169, 382170
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, extracting one or more color from the image and modifying the video stream. The systems, devices, and method convert images of the set of images to a coordinate representation, perform a histogram equalization, identifies one or more colors of the coordinate representation based on an area of interest, determines a prevailing color, and applies the prevailing color to pixels of the video stream.

Claims

1. A computer implemented method, comprising:

receiving, by one or more processors, a set of images of a video stream, the set of images depicting at least a portion of a face within the video stream;

converting, by the one or more processors, a first portion of images of the set of images to a set of cylindrical coordinate representations as the set of images is being captured;

performing a first histogram equalization on the set of cylindrical coordinate representations;

detecting an area of interest for the first portion of images of the set of images, the area of interest positioned on the portion of the face within the video stream;

identifying one or more colors of the set of cylindrical coordinate representations based on the area of interest, the one or more colors identified within the portion of the face;

determining a prevailing color of the one or more colors within the area of interest;

comparing pixel values within the area of interest to a set of predetermined pixel value ranges selected from a plurality of sets of pixel value ranges each associated with an object type;

converting the prevailing color from the cylindrical coordinate representation to a color space associated with the set of images of the video stream while the set of images are being captured;

applying the prevailing color to a second portion of images of the set of images of the video stream while the set of images are being captured; and

generating a modified set of images by performing a second histogram equalization on the set of cylindrical coordinate representations of the second portion of images after applying the prevailing color.

2. The method of claim 1, wherein the first histogram equalization further comprises:

identifying a first range of color values for a set of color channels of the set of images; and

translating the first range of color values to a first uniform color range.

3. The method of claim 2, wherein the first histogram equalization further comprises:

identifying one or more gray pixels within the set of cylindrical coordinate representations;

determining a lowest gray value among the one or more gray pixels;

assigning the lowest gray value as an absolute gray value to generate a color shift; and

applying a color shift to one or more color channels of the set of color channels to generate one or more color shifted images.

4. The method of claim 3, further comprising:

converting one or more pixels within the area of interest from a first color space to a second color space to generate one or more converted pixels;

identifying average pixel values for the one or more converted pixels;

comparing the pixel values to a set of predetermined pixel value ranges; and

modifying the pixel values to generate a set of modified pixel values within the set of predetermined pixel value ranges.

5. The method of claim 4 further comprising:

modifying pixel values for a set of pixels outside of the area of interest based on the set of modified pixel values to generate one or more modified images;

identifying a second range of color values for a set of color channels of the one or more modified images; and

translating the second range of color values to a second uniform color range.

6. The method of claim 1 further comprising:

identifying an upper threshold for a lightness value within the set of cylindrical coordinate representations;

determining one or more pixel values exceeding the upper threshold; and

removing the one or more pixel values exceeding the upper threshold.

7. The method of claim 1 further comprising:

identifying a lower threshold for a lightness value within the set of cylindrical coordinate representations;

determining one or more pixel values which are below the lower threshold; and

removing the one or more pixel values below the lower threshold.

8. The method of claim 1 further comprising:

identifying one or more regions of interest within the set of images of the video stream;

tracking the one or more regions of interest; and

applying the prevailing color within the one or more regions of interest within the set of images of the video stream.

9. A system, comprising:

one or more processors; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by the one or more processors, a set of images of a video stream, the set of images depicting at least a portion of a face within the video stream;

converting, by the one or more processors, a first portion of images of the set of images to a set of cylindrical coordinate representations as the set of images is being captured;

performing a first histogram equalization on the set of cylindrical coordinate representations;

detecting an area of interest for the first portion of images of the set of images, the area of interest positioned on the portion of the face within the video stream;

identifying one or more colors of the set of cylindrical coordinate representations based on the area of interest, the one or more colors identified within the portion of the face;

determining a prevailing color of the one or more colors within the area of interest;

comparing pixel values within the area of interest to a set of predetermined pixel value ranges selected from a plurality of sets of pixel value ranges each associated with an object type;

converting the prevailing color from the cylindrical coordinate representation to a color space associated with the set of images of the video stream while the set of images are being captured;

applying the prevailing color to a second portion of images of the set of images of the video stream while the set of images are being captured; and

generating a modified set of images by performing a second histogram equalization on the set of cylindrical coordinate representations of the second portion of images after applying the prevailing color.

10. The system of claim 9, wherein the first histogram equalization further comprises

identifying a first range of color values for a set of color channels of the set of images;

and

translating the first range of color values to a first uniform color range.

11. The system of claim 10, wherein the first histogram equalization further comprises:

identifying one or more gray pixels within the set of cylindrical coordinate representations;

determining a lowest gray value among the one or more gray pixels;

assigning the lowest gray value as an absolute gray value to generate a color shift; and

applying a color shift to one or more color channels of the set of color channels to generate one or more color shifted images.

12. The system of claim 11, wherein the operations further comprise:

converting one or more pixels within the area of interest from a first color space to a second color space to generate one or more converted pixels;

identifying average pixel values for the one or more converted pixels;

comparing the pixel values to a set of predetermined pixel value ranges; and

modifying the pixel values to generate a set of modified pixel values within the set of predetermined pixel value ranges.

13. The system of claim 12, wherein the operations further comprise:

modifying pixel values for a set of pixels outside of the area of interest based on the set of modified pixel values to generate one or more modified images;

identifying a second range of color values for a set of color channels of the one or more modified images; and

translating the second range of color values to a second uniform color range.

14. The system of claim 9 further comprising:

identifying an upper threshold for a lightness value within the set of cylindrical coordinate representations;

determining one or more pixel values exceeding the upper threshold;

identifying a lower threshold for a lightness value within the set of cylindrical coordinate representations;

determining one or more pixel values which are below the lower threshold;

removing the one or more pixel values exceeding the upper threshold; and

removing the one or more pixel values below the lower threshold.

15. The system of claim 9, wherein the operations further comprise:

identifying one or more regions of interest within the set of images of the video stream;

tracking the one or more regions of interest; and

applying the prevailing color within the one or more regions of interest within the set of images of the video stream.

16. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, by one or more processors, a set of images of a video stream, the set of images depicting at least a portion of a face within the video stream;

converting, by the one or more processors, a first portion of images of the set of images to a set of cylindrical coordinate representations as the set of images is being captured;

performing a first histogram equalization on the set of cylindrical coordinate representations;

detecting an area of interest for the first portion of images of the set of images, the area of interest positioned on the portion of the face within the video stream;

identifying one or more colors of the set of cylindrical coordinate representations based on the area of interest, the one or more colors identified within the portion of the face;

determining a prevailing color of the one or more colors within the area of interest;

comparing pixel values within the area of interest to a set of predetermined pixel value ranges selected from a plurality of sets of pixel value ranges each associated with an object type;

converting the prevailing color from the cylindrical coordinate representation to a color space associated with the set of images of the video stream while the set of images are being captured;

applying the prevailing color to a second portion of images of the set of images of the video stream while the set of images are being captured; and

generating a modified set of images by performing a second histogram equalization on the set of cylindrical coordinate representations of the second portion of images after applying the prevailing color.

17. The non-transitory machine-readable storage medium of claim 16, wherein the first histogram equalization further comprises:

identifying a first range of color values for a set of color channels of the set of images; and

translating the first range of color values to a first uniform color range.

18. The non-transitory machine-readable storage medium of claim 17, wherein the first histogram equalization further comprises:

identifying one or more gray pixels within the set of cylindrical coordinate representations;

determining a lowest gray value among the one or more gray pixels;

assigning the lowest gray value as an absolute gray value to generate a color shift; and

applying a color shift to one or more color channels of the set of color channels to generate one or more color shifted images.

19. The non-transitory machine-readable storage medium of claim 18, wherein the operations further comprise:

converting one or more pixels within the area of interest from a first color space to a second color space to generate one or more converted pixels;

identifying average pixel values for the one or more converted pixels;

comparing the pixel values to a set of predetermined pixel value ranges; and

modifying the pixel values to generate a set of modified pixel values within the set of predetermined pixel value ranges.

20. The non-transitory machine-readable storage medium of claim 19, wherein the operations further comprise:

modifying pixel values for a set of pixels outside of the area of interest based on the set of modified pixel values to generate one or more modified images;

identifying a second range of color values for a set of color channels of the one or more modified images; and

translating the second range of color values to a second uniform color range.