System and method to digitally replace objects in images or video

• US 10,249,093 B2
• Filed: 03/13/2017
• Issued: 04/02/2019
• Est. Priority Date: 12/05/2011
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Zavesky, Eric
• Primary Examiner(s): Bailey, Frederick D
• Application Number:
  US15/457,564
• Publication Number:
  US 20170186241A1
• Time in Patent Office: 750 Days
• Field of Search: None
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A method includes receiving video data representative of a video frame. The video frame includes a first representation of a first object. The method also includes determining whether to replace the first representation in the video frame with a second representation of a second object based on an object matching rule. In response to determining that the first representation is to be replaced with the second representation, the method includes manipulating a three-dimensional model of the second object to generate the second representation of the second object so that a first visual property of the second representation matches a second visual property of the first representation. The method further includes generating a modified version of the video frame that includes the second representation in place of the first representation.

Claims

1. A method comprising:

receiving video data representative of a video frame, the video frame including a first representation of a first object;

and

in response to determining that the first representation is to be replaced with a second representation of a second object;

identifying a first visual property of the first representation;

manipulating a three-dimensional model of the second object to generate the second representation of the second object so that a second visual property of the second representation matches the first visual property of the first representation; and

generating a modified version of the video frame that includes the second representation in place of the first representation.

2. The method of claim 1, further comprising determining whether to replace the first representation with the second representation based on an object matching rule.

3. The method of claim 2, wherein the object matching rule specifies audio criteria, visual criteria, lighting criteria, texture criteria, geometric criteria, or any combination thereof.

4. The method of claim 3, wherein the object matching rule specifies that the first representation of the first object is not to be replaced with the second representation of the second object when a third representation of a particular third object is present in the video frame.

5. The method of claim 2, wherein the object matching rule identifies a plurality of object measurements.

6. The method of claim 2, wherein the object matching rule identifies a particular orientation of the first representation, and wherein the first representation is to be replaced with the second representation responsive to the first representation having an orientation different than the particular orientation.

7. The method of claim 2, wherein determining whether to replace the first representation with the second representation is performed in response to a third visual property of the first representation failing to satisfy a requirement, and wherein the second representation satisfies the requirement.

8. The method of claim 7, wherein the object matching rule is based on the requirement, and wherein the requirement includes a lighting requirement, a texture requirement, or a combination thereof.

9. The method of claim 1, further comprising computing the three-dimensional model of the second object based on a media content item that depicts the second object and storing data representing the three-dimensional model in a database.

10. The method of claim 9, wherein the media content item comprises a graphics file, a video file, or a combination thereof.

11. The method of claim 9, wherein computing the three-dimensional model of the second object includes extrapolating a portion of the three-dimensional model based on partial models of the second object, two-dimensional models of the second object, scene data depicting the second object, multi-view images of the second object, motion data related to the second object in a scene, or any combination thereof.

12. The method of claim 11, wherein the scene data includes data associated with visual content surrounding the second object, audio content in the media content item, or any combination thereof.

13. A system comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured to;

determine whether to replace a first representation of a first object in a video frame with a second representation of a second object based on an object matching rule; and

in response to determining that the first representation is to be replaced with the second representation;

identifying a first visual property of the first representation;

manipulating a three-dimensional model of the second object to generate the second representation of the second object so that a second visual property of the second representation matches the first visual property of the first representation; and

generate a modified version of the video frame that includes the second representation in place of the first representation.

14. The system of claim 13, wherein the processor is further configured to extrapolate the three-dimensional model of the second object based on a media content item that depicts the second object and to store the three-dimensional model in the memory.

15. The system of claim 14, wherein the media content item comprises a graphics file, a video file, or a combination thereof.

16. The system of claim 13, wherein the object matching rule indicates a threshold number, and wherein the processor is configured to replace the first representation with the second representation in response to a number of features of the second object that match features of the first object satisfying the threshold number.

17. The system of claim 13, wherein the object matching rule comprises a user defined set of matching rules, and wherein the processor is configured to replace the first representation with the second representation in response to the user defined set of matching rules being satisfied.

18. The system of claim 13, wherein manipulating the three-dimensional model includes rotating the three-dimensional model, changing a scale of the three-dimensional model, applying a bi-directional reflectance distribution function to the three-dimensional model, applying a texture to the three-dimensional model, or a combination thereof.

19. A computer-readable storage device comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:

determining whether to replace a first representation of a first object in a video frame with a second representation of a second object based on an object matching rule; and

in response to determining that the first representation is to be replaced with the second representation;

identifying a first visual property of the first representation;

manipulating a three-dimensional model of the second object to generate the second representation of the second object so that a second visual property of the second representation matches the first visual property of the first representation; and

generating a modified version of the video frame that includes the second representation in place of the first representation.

20. The computer-readable storage device of claim 19, wherein the operations further comprise:

computing the three-dimensional model of the second object based on a media content item that depicts the second object; and

storing the three-dimensional model in a memory.