NETWORKED CAPTURE AND 3D DISPLAY OF LOCALIZED, SEGMENTED IMAGES

• US 20140233847A1
• Filed: 09/12/2011
• Published: 08/21/2014
• Est. Priority Date: 09/12/2011
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Ratcliff, Joshua, Wu, Yi, Choubassi, Maha El, Sun, Wei, Seshadrinathan, Kalpana, Kozintsev, Igor V., Gat, Yoram
• Granted Patent: US 9,418,438 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Systems, devices and methods are described including receiving a source image having a foreground portion and a background portion, where the background portion includes image content of a three-dimensional (3D) environment. A camera pose of the source image may be determined by comparing features of the source image to image features of target images of the 3D environment and using the camera pose to segment the foreground portion from the background portion may generate a segmented source image. The resulting segmented source image and the associated camera pose may be stored in a networked database. The camera pose and segmented source image may be used to provide a simulation of the foreground portion in a virtual 3D environment.

Claims

1-30. -30. (canceled)

31. A method for segmenting images, comprising:

receiving a source image, the source image including a foreground portion and a background portion, wherein the background portion includes image content of a three-dimensional (3D) environment;

determining a camera pose of the source image by comparing the source image to a plurality of target images of the 3D environment;

generating a segmented source image by segmenting the foreground portion from the background portion in response to the camera pose; and

storing the segmented source image and the camera pose in a networked database.

32. The method of claim 31, wherein determining the camera pose comprises:

extracting at least one source image visual feature from the source image;

determining a best matched target image by comparing the at least one source image visual feature to the plurality of target images; and

projecting the source image onto a planar structure in the 3D environment associated with the best-matched target image, the planar structure having 3D coordinates in the 3D environment.

33. The method of claim 31, wherein segmenting the foreground portion comprises:

for each target image, applying a homography matrix to the target image to generate a projection mask and a projected target image, the projection mask indicating at least some of the background portion of the target image within the source image;

color adjusting each projected target image to the source image;

generating a plurality of difference maps by comparing the color-adjusted projected target images to the source image; and

merging the difference maps to generate a blended difference map.

34. The method of claim 33, further comprising:

applying dual-thresholding to the blended difference map to generate a plurality of first binary foreground masks; and

refining each first binary foreground mask by performing statistical analysis of the source image and the color-adjusted projected target image.

35. The method of claim 33, further comprising:

subtracting floor portions from the filtered source image to produce a floor mask;

blending the plurality of first refined foreground masks and the floor mask to produce a second binary foreground mask; and

applying a median filter to the second binary foreground mask.

36. The method of claim 35, further comprising:

detecting face regions in the second binary foreground mask to generate a face mask;

blending the face mask with the second binary foreground mask to generate a third binary foreground mask; and

applying connected component analysis to the third binary foreground mask.

37. The method of claim 33, wherein color adjusting each projected target image to the source image comprises:

generating a difference map by comparing each projected target image to the source image;

applying a fixed threshold to the difference map to generate a coarsely segmented foreground mask;

subtracting the coarsely segmented foreground mask from the projection mask to generate the adjusted projection mask;

estimating a color mapping, within the adjusted projection mask, from the projected target image to the source image; and

applying the color mapping, within the original projection mask, to the projected target image to generate the color-adjusted projected target image.

38. The method of claim 31, further comprising:

using the camera pose and segmented source image to provide a simulation of the foreground portion in the 3D environment.

39. A system, comprising:

a processor and a memory coupled to the processor, wherein instructions in the memory configure the processor to;

receive a source image, the source image including a foreground portion and a background portion, wherein the background portion includes image content of a three-dimensional (3D) environment;

determine a camera pose of the source image by comparing the source image to a plurality of target images of the 3D environment;

generate a segmented source image by segmenting the foreground portion from the background portion in response to the camera pose; and

store the segmented source image and the camera pose in a networked database.

40. The system of claim 39, wherein instructions that configure the processor to determine the camera pose include instructions that configure the processor to:

extract at least one source image visual feature from the source image;

determine a best matched target image by comparing the at least one source image visual feature to the plurality of target images; and

project the source image onto a planar structure in the 3D environment associated with the best-matched target image, the planar structure having 3D coordinates in the 3D environment.

41. The system of claim 39, wherein instructions that configure the processor to segment the foreground portion include instructions that configure the processor to:

for each target image, apply a homography matrix to the target image to generate a projection mask and a projected target image, the projection mask indicating at least some of the background portion of the target image within the source image;

color adjust each projected target image to the source image;

generate a plurality of difference maps by comparing the color-adjusted projected target images to the source image; and

merge the difference maps to generate a blended difference map.

42. The system of claim 41, further comprising instructions that configure the processor to:

apply dual-thresholding to the blended difference map to generate a plurality of first binary foreground masks; and

refine each first binary foreground mask by performing statistical analysis of the source image and the color-adjusted projected target image.

43. The system of claim 42, further comprising instructions that configure the processor to:

subtract floor portions from the filtered source image to produce a floor mask;

blend the plurality of first refined foreground masks and the floor mask to produce a second binary foreground mask; and

apply a median filter to the second binary foreground mask.

44. The system of claim 43, further comprising instructions that configure the processor to:

detect face regions in the second binary foreground mask to generate a face mask;

blend the face mask with the second binary foreground mask to generate a third binary foreground mask; and

apply connected component analysis to the third binary foreground mask.

45. The system of claim 41, wherein instructions that configure the processor to color adjust each projected target image to the source image include instructions that configure the processor to:

generate a difference map by comparing each projected target image to the source image;

apply a fixed threshold to the difference map to generate a coarsely segmented foreground mask;

subtract the coarsely segmented foreground mask from the projection mask to generate the adjusted projection mask;

estimate a color mapping, within the adjusted projection mask, from the projected target image to the source image; and

apply the color mapping, within the original projection mask, to the projected target image to generate the color-adjusted projected target image.

46. The system of claim 39, further comprising instructions that configure the processor to:

use the camera pose and segmented source image to provide a simulation of the foreground portion in the 3D environment.

47. An article comprising a computer program product having stored therein instructions that, if executed, result in:

receiving a source image, the source image including a foreground portion and a background portion, wherein the background portion includes image content of a three-dimensional (3D) environment;

determining a camera pose of the source image by comparing the source image to a plurality of target images of the 3D environment;

generating a segmented source image by segmenting the foreground portion from the background portion in response to the camera pose; and

storing the segmented source image and the camera pose in a networked database.

48. The article of claim 47, wherein the instructions that result in determining the camera pose include instructions that, if executed, result in:

extracting at least one source image visual feature from the source image;

determining a best matched target image by comparing the at least one source image visual feature to the plurality of target images; and

projecting the source image onto a planar structure in the 3D environment associated with the best-matched target image, the planar structure having 3D coordinates in the 3D environment.

49. The article of claim 47, wherein the instructions that result in segmenting the foreground portion include instructions that, if executed, result in:

for each target image, applying a homography matrix to the target image to generate a projection mask and a projected target image, the projection mask indicating at least some of the background portion of the target image within the source image;

color adjusting each projected target image to the source image;

generating a plurality of difference maps by comparing the color-adjusted projected target images to the source image; and

merging the difference maps to generate a blended difference map.

50. The article of claim 49, further comprising instructions that, if executed, result in:

applying dual-thresholding to the blended difference map to generate a plurality of first binary foreground masks; and

refining each first binary foreground mask by performing statistical analysis of the source image and the color-adjusted projected target image.

51. The article of claim 50, further comprising instructions that, if executed, result in:

subtracting floor portions from the filtered source image to produce a floor mask;

blending the plurality of first refined foreground masks and the floor mask to produce a second binary foreground mask; and

applying a median filter to the second binary foreground mask.

52. The article of claim 51, further comprising instructions that, if executed, result in:

detecting face regions in the second binary foreground mask to generate a face mask;

blending the face mask with the second binary foreground mask to generate a third binary foreground mask; and

applying connected component analysis to the third binary foreground mask.

53. The article of claim 49, wherein the instructions that result in color adjusting each projected target image to the source image include instructions that, if executed, result in:

generating a difference map by comparing each projected target image to the source image;

applying a fixed threshold to the difference map to generate a coarsely segmented foreground mask;

subtracting the coarsely segmented foreground mask from the projection mask to generate the adjusted projection mask;

estimating a color mapping, within the adjusted projection mask, from the projected target image to the source image; and

applying the color mapping, within the original projection mask, to the projected target image to generate the color-adjusted projected target image.

54. The article of claim 47, further comprising instructions that, if executed, result in:

using the camera pose and segmented source image to provide a simulation of the foreground portion in the 3D environment.