System and method to digitally replace objects in images or video

• US 9,626,798 B2
• Filed: 12/05/2011
• Issued: 04/18/2017
• Est. Priority Date: 12/05/2011
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Zavesky, Eric
• Primary Examiner(s): Bailey, Frederick
• Application Number:
  US13/311,044
• Publication Number:
  US 20130141530A1
• Time in Patent Office: 1,961 Days
• Field of Search: None
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A method includes receiving video data and identifying a second object in at least one video frame of the video data. The method also includes determining whether to replace the second object in the at least one video frame with a first object based on at least one object matching rule. In response to determining that the second object is to be replaced with the first object, the method includes manipulating the three-dimensional model of the first object to generate a representation of the first object that matches at least one visual property of the second object and replacing the second object with the representation of the first object in the at least one video frame.

Claims

1. A method comprising:

receiving video data;

identifying a second object depicted in a video frame of the video data based on a comparison of the second object as depicted in the video frame to a set of three-dimensional object models;

determining whether to replace the second object in the video frame with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object;

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frame.

2. The method of claim 1, wherein the object matching rule comprises a threshold number of matching features of the first object and the second object, a user defined set of matching rules, a context rule, or any combination thereof.

3. The method of claim 2, wherein the context rule specifies audio criteria, visual criteria, lighting criteria, texture criteria, geometric criteria, or any combination thereof.

4. The method of claim 3, wherein the context rule specifies that the second object is not to be replaced with the representation of the first object when a particular third object is present in the video frame.

5. The method of claim 2, wherein the matching features of the second object include color of the second object, size of the second object, shape of the second object, or any combination thereof.

6. The method of claim 1, further comprising automatically performing pixel hallucination for missing pixels resulting from the replacement of the second object with the representation of the first object.

7. The method of claim 1, further comprising computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object and storing the three-dimensional model in a database.

8. The method of claim 7, wherein the media content item comprises a graphics file, a video file, an automatically crawled network accessible media content item, a user provided media content item, or any combination thereof.

9. The method of claim 7, further comprising computing the three-dimensional model of the first object based on partial models of the first object, two-dimensional models of the first object, scene data depicting the first object, multi-view images of the first object, motion of the first object in a scene, or any combination thereof.

10. The method of claim 7, further comprising extracting scene data associated with the first object, wherein the scene data includes data regarding visual content surrounding the first object in the media content item, audio content in the media content item, or any combination thereof.

11. The method of claim 10, further comprising:

extracting the audio content via speech recognition; and

associating the first object with a particular scene based on the visual content and the audio content.

12. The method of claim 1, wherein the visual property comprises shape, texture, shade, topology, or any combination thereof.

13. A system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform operations including;

identifying a second object depicted in video frames of video data based on a comparison of the second object as depicted in the video frames to a set of three-dimensional object models;

determining whether to replace the second object in the video frames with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object;

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frames.

14. The system of claim 13, wherein the operations further include:

computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object; and

storing the three-dimensional model in the memory.

15. The system of claim 14, wherein the media content item comprises a graphics file, a video file, an automatically crawled network accessible media content item, a user provided media content item, or any combination thereof.

16. The system of claim 13, wherein the object matching rule comprises a threshold number of matching features of the first object and the second object.

17. The system of claim 13, wherein the visual property comprises a bi-directional reflectance distribution function of the second object.

18. The system of claim 13, wherein the requirement indicates a particular feature of a particular object is visible, and wherein the three-dimensional model is manipulated so that the particular feature of the first object is visible in the video frames.

19. A computer-readable storage device comprising instructions that, when executed by a processor, cause the processor to perform operations including:

identifying a second object depicted in a video frame of video data based on a comparison of the second object as depicted in the video frame to a set of three-dimensional object models;

determining whether to replace the second object in the video frame with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object;

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frame.

20. The computer-readable storage device of claim 19, wherein the operations further include:

computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object; and

storing the three-dimensional model in a memory.