IMAGE PROCESSING

• US 20150248775A1
• Filed: 10/02/2013
• Published: 09/03/2015
• Est. Priority Date: 10/03/2012
• Status: Active Grant

• Current Assignee: Holition Limited
• Original Assignee: Holition Limited
• Inventors: Freeman, Russell
• Granted Patent: US 9,552,655 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

There is described an image processing method in which a scene is repeatedly imaged to form a series of input images. For at least a subset of the input images, a colour calibration procedure is conducted which populates a foreground colour histogram with the frequency of occurrence of colour values in a stencil area of the input image, and populates a background colour histogram with the frequency of occurrence of colour values outside of the stencil portion of the input image. For at least a subset of the input images, a colour replacement procedure is conducted which updates the stencil area based on a determination, from the colour values of pixels within the input image, of likelihood values representing the likelihood of pixels belonging to an image area of interest, the likelihood value for each colour value being determined from a combination of the foreground and background colour histograms, replaces the original colour values of pixels within the updated stencil area of the input image with replacement colour values, and displays the image on which colour replacement processing has been conducted. In this way, a stencil area is determined based on foreground/background histogramming, and used both to define an area to which colour replacement processing is to be conducted, and also an area for use in further populating the colour histograms to calibrate the colour replacement processing.

Claims

1-28. -28. (canceled)

29. An image processing method, comprising the steps of:

imaging a scene to form an input image;

generating a depth map of the scene;

determining, from the depth map, an area of interest in the input image to which colour replacement processing can be applied;

identifying pixels within the area of interest of the input image having a colour value within a specified colour range;

replacing the colour value of each identified pixel with a replacement colour value;

generating an output image in which the replacement colour values are used for the identified pixels; and

displaying the output image.

30. An image processing method according to claim 29, wherein the area of interest determined from the depth map is a foreground area of the image.

31. An image processing method according to claim 30, comprising a step of:

calibrating the specified colour range to the colour distribution of an area of interest in the input image;

wherein the area of interest used for colour calibration is based on the foreground area determined from the depth map.

47. The method of claim 29, further comprising:

repeatedly imaging a scene to form a series of input images;

for at least a subset of the input images, conducting a colour calibration procedure of;

populating a foreground colour histogram with the frequency of occurrence of colour values in a stencil area of the input image; and

populating a background colour histogram with the frequency of occurrence of colour values outside of the stencil portion of the input image; and

for at least a subset of the input images, conducting a colour replacement procedure of;

updating the stencil area based on a determination, from the colour values of pixels within the input image, of likelihood values representing the likelihood of pixels belonging to an image area of interest, the likelihood value for each colour value being determined from a combination of the foreground and background colour histograms;

replacing the original colour values of pixels within the updated stencil area of the input image with replacement colour values; and

displaying the image on which colour replacement processing has been conducted.

48. An image processing method according to claim 47, wherein:

the stencil area is initialised as a mouth area of the face of a person within the scene being imaged; and

the background colour histogram is populated with the frequency of occurrence of colour values outside of the stencil portion of the input image but within a bounded area surrounding the stencil portion, the bounded area being initialised as a face area of the person within the scene being imaged.

49. The method of claim 29, further comprising calibrating a depth sensor with a first camera, the depth sensor being provided with a second camera pre-calibrated to the depth sensor, by:

imaging a scene in which a predetermined marker is visible using the first camera to form a first input image;

imaging the scene using the second camera to form a second input image;

detecting the position and orientation of the predetermined marker within the first input image and the second image respectively;

calculating a transform representing a difference in alignment between the first input image and the second input image based on the difference in the detected position and orientation of the predetermined marker within the first input image and the second input image; and

mapping depth information captured by the depth sensor to image information captured by the first camera using the calculated transform.

32. An image processing apparatus, comprising:

a video camera operable to image a scene to form an input image;

a depth sensor operable to generate a depth map of the scene;

a processor, said processor configured to;

determine, from the depth map, an area of interest in the input image to which colour alteration processing may be applied;

identify pixels within the area of interest in the input image having a colour value within a specified colour range;

replace the colour value of each identified pixel with a replacement colour value; and

generate an output image in which the replacement colour values are used for the identified pixels; and

a display device operable to display the output image.

33. An image processing apparatus according to claim 32, wherein the area of interest determined from the depth map is a foreground area of the image.

34. An image processing apparatus according to claim 33, wherein the processor is configured to calibrate the specified colour range to the colour distribution of an area of interest in the input image;

wherein the area of interest used for colour calibration is based on the foreground area determined from the depth map.

35-46. -46. (canceled)

50. A non-transitory computer-readable medium comprising computer-executable instructions, that when executed perform the method of:

imaging a scene to form an input image;

generating a depth map of the scene;

determining, from the depth map, an area of interest in the input image to which colour replacement processing can be applied;

identifying pixels within the area of interest of the input image having a colour value within a specified colour range;

replacing the colour value of each identified pixel with a replacement colour value;

generating an output image in which the replacement colour values are used for the identified pixels; and

displaying the output image.