REPLACING IMAGE INFORMATION IN A CAPTURED IMAGE

• US 20090297061A1
• Filed: 05/30/2008
• Published: 12/03/2009
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: General Instrument Corporation (CommScope Holding Company, Inc.)
• Inventors: Mareachen, Russell D., Super, Boaz J., Yu, Tianli, Tang, Bei, Chai, Sek M.
• Granted Patent: US 8,073,243 B2
• Time in Patent Office: Days
• US Class Current: 382/285
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Described herein are systems and methods for expanding upon the single-distance-based background denotation to seamlessly replace unwanted image information in a captured image derived from an imaging application so as to account for a selected object'"'"'s spatial orientation to maintain an image of the selected object in the captured image.

Claims

1. A method for replacing image information in an image comprising:

obtaining an image of a scene;

obtaining a depth map of the scene;

defining a depth surface in the depth map, wherein the depth surface includes at least two different depth values;

defining at least one portion of the image based on the depth surface; and

replacing the at least one portion of the image with selected image information.

2. The method of claim 1, wherein the step of defining the depth surface comprises:

identifying a portion of the obtained image that corresponds to a selected object in the scene; and

mapping the portion of the obtained image that corresponds to the selected object in the scene to a set of coordinates in the depth map.

3. The method of claim 2, wherein the set of coordinates in the depth map are three-dimensional coordinates, and the step of defining the depth surface further comprises:

defining the depth surface based on at least one of;

the three-dimensional coordinates, an approximation of the three-dimensional coordinates, and an offset from the three-dimensional coordinates.

4. The method of claim 1, wherein the step of defining the depth surface comprises at least one of:

determining a parametric surface model for the depth surface; and

determining a non-parametric surface model for the depth surface.

5. The method of claim 1, wherein the depth surface includes a plane that is not parallel to a principal plane of the obtained image.

6. The method of claim 2, wherein the depth surface comprises a representation of at least a portion of a surface of the selected object.

7. The method of claim 2, wherein the step of defining the depth surface comprises:

determining a surface of the selected object based on the mapping of the portion of the obtained image; and

fitting a surface model to the determined surface of the selected object.

8. The method of claim 7, wherein the step of defining the depth surface further comprises:

forming the depth surface by extending the surface model along an entire width direction and an entire height direction of the scene as mapped in the depth map.

9. The method of claim 2, wherein:

the step of mapping the portion of the obtained image includes dynamically mapping the portion of the obtained image that corresponds to the selected object as the selected object moves about in the scene; and

the step of defining the depth surface includes dynamically defining the depth surface based on the dynamic mapping of the selected objected in the scene.

10. The method of claim 1, wherein:

the step of defining the depth surface includes dynamically defining the depth surface based on changes in at least one of the obtained image, the obtained depth map, and the scene.

11. The method of claim 1, wherein the step of obtaining the image of the scene comprises:

obtaining the image of the scene with an imaging system that includes one of a normal lens, a wide-angle lens, a telephoto lens, and a macro lens.

12. The method of claim 1, wherein the step of defining at least one portion of the image comprises:

determining a background portion of the image by identifying pixels in the captured image that have depth values greater than depth values of the depth surface; and

determining a foreground portion of the image by identifying pixels in the captured image that have depth values less than the depth values of the depth surface.

13. The method of claim 1, further comprising:

defining a virtual replacement surface.

14. The method of claim 13, further comprising:

modifying properties of the selected image information based on depth values of the virtual replacement surface.

15. The method of claim 13, further comprising:

modifying properties of the selected image information based on distances of the virtual replacement surface to the depth surface.

16. The method of claim 1, further comprising:

modifying properties of the selected image information based on depth values of the depth surface.

17. The method of claim 1, further comprising:

modifying properties of the selected image information so as to match a gradient of the selected image information to a gradient of the at least one portion of the image.

18. The method of claim 1, wherein the image obtained is a video image.

19. The method of claim 1, wherein the steps of obtaining the image of the physical scene and obtaining the depth map of the physical scene are performed using a stereo camera.

20. The method of claim 1, wherein the depth surface includes non-contiguous zones.

21. The method of claim 13, wherein the virtual replacement surface includes non-contiguous zones.

22. A system to replace image information in an image comprising:

means for obtaining an image of a scene;

means for obtaining a depth map of the scene;

means for defining a depth surface in the depth map, wherein the depth surface includes at least two different depth values;

means for defining at least one portion of the image based on the depth surface; and

means for replacing the at least one portion of the image with selected image information.

23. A computer readable medium on which is encoded machine-executable instructions that are executable to:

obtain an image of a scene;

obtain a depth map of the scene;

define a depth surface in the depth map, wherein the depth surface includes at least two different depth values;

define at least one portion of the image based on the depth surface; and

replace the at least one portion of the image with selected image information.