System and method to digitally replace objects in images or video

• US 10,580,219 B2
• Filed: 02/05/2019
• Issued: 03/03/2020
• Est. Priority Date: 12/05/2011
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Zavesky, Eric
• Primary Examiner(s): Bailey, Frederick D
• Application Number:
  US16/267,649
• Publication Number:
  US 20190172268A1
• Time in Patent Office: 392 Days
• Field of Search: None
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A method includes receiving, at a computer system, video data of a video frame of video content. The method includes identifying, at the computer system, a first representation of a first object in the video frame to be replaced with a second representation of a second object based on absence of a visible feature in the first representation. The method includes manipulating, at the computer system, a three-dimensional model of the second object to generate the second representation. The second representation includes the visible feature. The method also includes generating, at the computer system, a modified version of the video frame that includes the second representation in place of the first representation.

Claims

1. A method comprising:

receiving, at a computer system, video data of a video frame of video content;

identifying, at the computer system, a first representation of a first object in the video frame to be replaced with a second representation of a second object, wherein the first representation is to be replaced based on absence of a visible feature in the first representation;

manipulating, at the computer system, a three-dimensional model of the second object to generate the second representation, wherein the second representation includes the visible feature; and

generating, at the computer system, a modified version of the video frame that includes the second representation in place of the first representation.

2. The method of claim 1, wherein generating the modified version of the video frame includes removing the first representation from the video frame.

3. The method of claim 2, wherein generating the modified version of the video frame further includes inserting the second representation to fill at least a portion of the video frame previously occupied by the first representation.

4. The method of claim 3, wherein generating the modified version of the video frame further comprises using pixel hallucination to fill a first portion of the video frame previously occupied by the first representation and not occupied by the second representation.

5. The method of claim 4, wherein pixels for the first portion are obtained from the video frame prior to removal of the first representation, a previous video frame of the video content, or a subsequent video frame of the video content.

6. The method of claim 1, wherein generating the modified version comprises:

determining that a reflection of the first object is present in another portion of the video frame; and

replacing the reflection of the first object with pixels corresponding to a reflection of the second object.

7. The method of claim 1, wherein manipulating the three-dimensional model of the second object includes adjusting a property of the three-dimensional model, the property including a color, size, shape, texture, lighting, shade, bi-directional reflectance distribution function, orientation, topology, or any combination thereof.

8. The method of claim 1, wherein manipulating the three-dimensional model of the second object comprises matching a particular visible feature of the second representation to match a corresponding visible feature of the first representation.

9. The method of claim 8, wherein the particular visible feature comprises a height of the second object, a width of the second object, a depth of the second object, or combinations thereof.

10. The method of claim 1, wherein the visible feature comprises at least a portion of a brand logo.

11. A system comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured to perform operations, the operations comprising;

receiving video data of a video frame of video content;

identifying a first representation of a first object in the video frame to be replaced with a second representation of a second object, the first representation to be replaced based on absence of a visible feature in the first representation;

manipulating a three-dimensional model of the second object to generate the second representation, wherein the second representation includes the visible feature; and

generating a modified version of the video frame that includes the second representation in place of the first representation.

12. The system of claim 11, wherein the processor is configured to:

extrapolate the three-dimensional model of the second object based on a media content item that depicts the second object; and

store the three-dimensional model in the memory.

13. The system of claim 12, wherein the media content item comprises a graphics file, a video file, or a combination thereof.

14. The system of claim 11, wherein manipulating the three-dimensional model includes rotating the three-dimensional model, changing a scale of the three-dimensional model, applying a texture to the three-dimensional model, or a combination thereof.

15. The system of claim 11, wherein the operations further comprise adjusting a bi-directional reflectance distribution function of the three-dimensional model to correspond to a particular bi-directional reflectance distribution function of the first representation.

16. A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to perform operations, the operations comprising:

receiving video data of a video frame of video content;

identifying a first representation of a first object in the video frame to be replaced with a second representation of a second object, the first representation to be replaced based on absence of a visible feature in the first representation;

manipulating a three-dimensional model of the second object to generate the second representation, wherein the second representation includes a visible feature; and

generating a modified version of the video frame that includes the second representation in place of the first representation.

17. The computer-readable storage device of claim 16, wherein the operations further comprise:

computing the three-dimensional model of the second object based on a media content item that depicts the second object; and

storing the three-dimensional model in a memory.

18. The computer-readable storage device of claim 16, wherein the operations further comprise sending the modified version to a user device.

19. The computer-readable storage device of claim 18, wherein the user device comprises a set-top box device, a media device, a mobile device, a personal computer, or combinations thereof.

20. The computer-readable storage device of claim 16, wherein generating the modified version further comprises:

removing the first representation from the video frame; and

inserting the second representation to fill a portion of the video frame previously occupied by the first representation.