Personalizing a Video

• US 20080019576A1
• Filed: 08/17/2007
• Published: 01/24/2008
• Est. Priority Date: 09/16/2005
• Status: Active Grant

• Current Assignee: Big Tent Entertainment, LLC
• Original Assignee: Flixor, Inc.
• Inventors: Shifflet, Tood, Senftner, Blake, Ralston, Liz, Lightwood, Miles
• Granted Patent: US 7,460,731 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Processes and apparatus for personalizing video through partial image replacement are disclosed. Personalization may include partial or full replacement of the image of an actor, an object or both. Personalization may also include insertion or replacement of an object, and full or partial replacement of the background and/or sound track. A video preparation process may be used to create a library of personalization-ready videos.

Claims

1. A process for personalizing an original digital video containing an image comprising an original background scene and a foreground including an original actor, the process comprising:

tracking the position, orientation, and expression of the original actor

replacing at least a key portion of the original actor with an image that continues the background scene

inserting a new actor into the video, the new actor substantially matching the position, orientation and expression of the replaced portion of the original actor

recreating illumination and shading effects on the new actor

recreating shadows and reflections of the new actor.

2. The process of claim 1, wherein the portion of the original actor that is replaced comprises at least the face and adjacent skin areas including visible portions of the ears and neck.

3. The process of claim 2, wherein

the image contains at least one skin area of the original actor that is not replaced, the skin area having a skin appearance that is different from a skin appearance of the new actor, and

the process further comprises altering the non-replaced skin area to match the skin appearance of the new actor.

4. The process of claim 1, wherein the entire original actor is replaced by an image that continues the background scene.

5. The process of claim 1, further comprising applying a combination and parameter morphing transformation process to the data that comprises the new actor, creating a hybrid new actor, prior to inserting.

6. The process of claim 1, wherein replacing further comprises replacing a shadow or a reflection of the original actor.

7. The process of claim 1, further comprising:

inserting a new object into the video

recreating illumination and shading effects on the new object

recreating shadows and reflections of the new object.

8. The process of claim 7, wherein inserting a new object uses a 3D model of the new object.

9. The process of claim 7, wherein the new object is selected based upon demographic information related to the new actor.

10. The process of claim 1,

wherein the original video contains an original object having a position and an orientation

the process further comprising:

replacing at least a portion of the original object with an image that continues the background scene

inserting a new object into the video, the new object substantially matching the position and orientation of the replaced portion of the original object

recreating illumination and shading effects on the new object

recreating shadows and reflections of the new object.

11. The process of claim 10, wherein inserting a new object uses a 3D model of the new object.

12. The process of claim 10, wherein the new object is selected based upon demographic information related to the new actor.

13. The process of claim 1, further comprising substituting a new background scene for at least a portion of the original background scene.

14. The process of claim 1, wherein

the video includes an audio track separable into an original background audio element and dialog spoken by the original actor

the process further comprising substituting a new background audio element for at least a portion of the original background audio element.

15. The process of claim 1, wherein

the video includes an audio track separable into an original background audio element and dialog spoken by the original actor

the process further comprising adding a new background audio element to the original audio track.

16. The process of claim 1, wherein the video includes an audio track separable into an original background audio element and dialog spoken by the original actor, the process further comprising:

obtaining a new actor voice sample

analyzing the new actor voice sample to define one or more descriptive features of the new actor'"'"'s voice

transforming the dialog spoken by the original actor using the one or more descriptive features of the new actor'"'"'s voice.

17. A process for personalizing a video, the process comprising:

providing a video library of a plurality of prepared videos, each of the prepared videos resulting from a video preparation process

providing an actor model library of one or more new actor models, each of the models resulting from an actor modeling process

selecting a video from the video library

selecting a new actor model from the actor model library

applying a personalization process to create a personalized version of the selected video using the selected new actor model.

18. The process of claim 17, the video preparation process further comprising:

providing a video containing an image comprising an original background scene and a foreground including an original actor

tracking the position, orientation, and expression of the original actor

replacing at least a key portion of the original actor with an image that continues the background scene

identifying and tracking illumination, shading, shadows, and reflections within the video.

19. The process of claim 18, the actor modeling process further comprising:

providing a least one 2D digital image of a new actor and related supporting information

creating a model of the new actor from the 2D digital image and related supporting information composed of a 3D model, a demographic profile and other personal information.

20. The process of claim 18, the personalization process further comprising:

using the new actor model to insert a new actor into a video, the new actor substantially matching the position, orientation and expression of the replaced portion of the original actor

recreating illumination and shading effects on the new actor

recreating shadows and reflections of the new actor.

21. The process of claim 20, wherein

the video contains at least one skin area of the original actor that is not replaced, the skin area having a skin appearance that is different from a skin appearance of the new actor, and

the process further comprises altering the non-replaced skin area to match the skin appearance of the new actor.

22. The process of claim 20, wherein:

the video preparation process further comprising;

tracking the position and orientation of an original object in the video

replacing at least a key portion of the original object with an image that continues the background scene

the personalization process further comprising;

inserting a new object into the video, the position, and orientation of the new object substantially matching the position and orientation of the original object

recreating illumination and shading effects on the new object

recreating shadows and reflections of the new object.

23. The process of claim 20, wherein:

the video preparation process further comprises;

tracking the position and orientation of a location suitable for placing an object within the video

the personalization process further comprises;

inserting a new object into the video, the position, and orientation of the new object substantially matching the position and orientation of the location

recreating illumination and shading effects on the new object

recreating shadows and reflections of the new object.

24. The process of claim 20, wherein the selected video includes an audio track separable into an original background soundtrack and dialog spoken by the original actor, and wherein:

the new actor modeling process further comprises;

obtaining a new actor voice sample

analyzing the new actor voice sample to define one or more descriptive features of the new actor'"'"'s voice

the personalization process further comprises;

morphing the dialog spoken by the original actor using the one or more descriptive features of the new actor'"'"'s voice.

25. A process for creating a personalized version of an original digital video, the original video containing an image of an original actor, the process comprising:

replacing at least a portion of the image of the original actor with an image of a new actor

inserting an image of a new object into the video.

26. The process of claim 25 wherein the image of the new object is selected from a plurality of images of candidate objects based upon demographic information related to the new actor.

27. The process of claim 26, the original video further containing an image of an original object, wherein at least a portion of the image of the original object is replaced by the image of the new object.

28. A computing device to create a personalized version of an original digital video containing an image comprising a background scene and a foreground including an image of an original actor, the computing device comprising:

a processor

a memory coupled with the processor

a storage medium having instructions stored thereon which when executed cause the computing device to perform actions comprising;

personalizing a video, the video containing an image including an image of an original actor, the actions further comprising;

tracking the position, orientation, and expression of the original actor

replacing at least a key portion of the original actor with an image that continues the background scene

inserting a new actor into the video, the new actor substantially matching the position, orientation and expression of the replaced portion of the original actor

recreating illumination and shading effects on the new actor

recreating shadows and reflections of the new actor.

29. The computing device of claim 28 further comprising an interface to a network, wherein the action performed by the computing device further comprises receiving a 2D digital image of the new actor via the network prior to the step of inserting.

30. The computing device of claim 29 wherein the action performed by the computing device further comprises transmitting the personalized video via the network after the step of recreating shadows and reflections.

31. The computing device of claim 29 further comprising an interface to one or more databases containing a plurality of videos.

32. The computing device of claim 31, wherein the action performed by the computing device prior to the step of substituting further comprises:

receiving a request to customize one of the plurality of videos via the network

retrieving the requested video from the one or more databases.

33. The computing device of claim 31, wherein the action performed by the computing device prior to the step of substituting further comprises:

selecting one of the plurality of videos based on demographic information related to the new actor

retrieving the selected video from the one or more databases.

34. The computing device of claim 28 further comprising a first interface to a digital image device, wherein the action performed by the computing device further comprises receiving a 2D digital image of the new actor via the interface prior to the step of inserting.

35. The computing device of claim 34 further comprising a second interface to a network, wherein the action performed by the computing device further comprises requesting and receiving the original video via the network prior to the step of substituting.

36. A storage medium having instructions stored thereon which when executed by a processor will cause the processor to perform actions comprising:

personalizing a video, the video containing an image including an image of an original actor, the actions further comprising;

tracking the position, orientation, and expression of the original actor

replacing at least a key portion of the original actor with an image that continues the background scene

inserting a new actor into the video, the new actor substantially matching the position, orientation and expression of the replaced portion of the original actor

recreating illumination and shading effects on the new actor

recreating shadows and reflections of the new actor.

37. A computer-implemented process for providing personalized digital video, comprising:

selecting a target in original digital video to be replaced by a target replacement, wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video;

analyzing each frame of the original digital video to track a change in the selected target in the original digital video to capture data on the selected target, wherein the captured data includes at least information on a position, orientation and size of the selected target in the original digital video; and

replacing the selected target with an image that resembles a continuation of a scene adjacent to the target in the original digital video to produce altered digital video in which the selected target is removed.

38. The process of claim 37, comprising:

storing the captured data of the selected target and the altered digital video in an association with each other in a digital library for producing personalized digital video.

39. The process of claim 37, wherein:

the selected target comprises a face of an actor in the original digital video, and

in analyzing each frame of the original digital video, an expression of the actor is tracked and captured as part of the data on the selected target.

40. The process of claim 37, comprising:

embedding a logic trigger in the personalized video to trigger logic that counts a number of times that the personalized digital video is played or a particular image in the personalized video is displayed.

41. The process of claim 40, wherein:

the logic includes a counter connected on the Internet to receive a trigger signal generated by a playback of the personalized digital video with the embedded logic trigger.

42. The process of claim 40, wherein:

the logic includes a counter which is located in a device that plays the personalized video.

43. The process of claim 42, comprising:

transmitting values of the counter to the Internet when the device is connected to the Internet.

44. The process of claim 40, comprising:

triggering the logic trigger when a specific product image or a logo in the personalized video is displayed.

45. The process of claim 40, wherein:

the logic is embedded in the personalized digital video.

46. The process of claim 45, wherein:

the logic includes a counter.

47. The process of claim 37, comprising:

inserting an image of the target replacement at a position of the selected target in each frame of the altered digital video in which the selected target appears in the original digital video based on the captured data of the selected target to substantially match at least the position, orientation and size of the selected target in the original digital video to produce a personalized digital video.

48. The process of claim 47, wherein:

inserting an image of the target replacement comprises;

applying the captured data of the selected target to a 3D model of the target replacement to produce a transformed 3D model of the target replacement in a frame that acquires characteristics of the selected target in the frame;

converting the transformed 3D model of the target replacement in a frame into a 2D image of the target replacement that is sized according to a size of the selected target in the frame; and

inserting the 2D image at a position of the selected target in the frame.

49. The process of claim 47, comprising:

in analyzing each frame of the original digital video, tracking and capturing information on illumination, shading, shadows and reflections associated with the selected target in the original digital video as part of the data on the selected target; and

after inserting the replacement target, using information on illumination, shading, shadows and reflections associated with the selected target in the original digital video to recreate the illumination, shading, shadows and reflections associated with the selected target in the original digital video for the target replacement in the personalized digital video.

71. The process of claim 37, wherein:

the original digital video includes a single frame image.

72. The process of claim 37, wherein:

the original digital video includes a collage style video sequence of frames, wherein each frame is a composite of a plurality of still images.

73. The process of claim 37, wherein:

the original digital video includes a video sequence of a plurality of still images.

74. The process of claim 37, wherein:

the altered digital video includes two or more selected targets that are replaced.

75. The process of claim 74, wherein:

a first selected target of the two or more selected targets is an actor in the original digital video, and

a second selected target of the two or more selected targets is a non-actor object in the original digital video.

76. The process of claim 37, comprising:

embedding a video identifier in the personalized digital video to allow for identification of the personalized digital video in a playback; and

embedding a target replacement identifier in the personalized digital video to allow for identification of the target replacement in a playback.

77. The process of claim 76, comprising:

using the video identifier to track a number of times the personalized video is played; and

using the target replacement identifier to track a number of times the user target replacement is displayed.

78. The process of claim 77, comprising:

embedding a logic trigger in the personalized video to trigger logic that counts a number of times that the user target replacement in the personalized video is displayed.

79. The process of claim 78, wherein:

the logic trigger includes a beginning logic trigger in a video frame where the user target replacement first appears in the personalized video and an ending trigger in a subsequent video frame where the user target replacement first disappears.

80. The process of claim 78, wherein:

the logic includes a counter connected on the Internet to receive a trigger signal generated by a playback of the personalized digital video with the embedded logic trigger.

81. The process of claim 78, wherein:

the logic includes a counter which is located in a device that plays the personalized video.

82. The process of claim 81, comprising:

transmitting values of the counter to the Internet when the device is connected to the Internet.

83. The process of claim 77, comprising:

storing information on the number of times the personalized video is played and the number of times the user target replacement is playeddisplayed in association with the video identifier for the personalized video and the target replacement identifier for the user target replacement.

84. The process of claim 83, wherein:

the user target replacement is an image associated with a product or service.

50. A computer-implemented process for providing personalized digital video, comprising:

storing altered digital video files based on original digital video files in a video library, wherein at least one target in an original video file is removed in a corresponding altered digital video file and is substituted by an image that resembles a continuation of a scene adjacent to the target in a frame of the original digital video file, and wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video file;

receiving, via a computer in communication with a network, a request from a user for personalizing a user selected altered digital video file by replacing a target in a corresponding original digital video file with a user target replacement identified by the user;

retrieving, from the video library, data on the target that is removed from the user selected altered digital video file, wherein the data includes at least information on a position, orientation and size of the target in the original digital video file for the user selected altered digital video file;

applying the retrieved data on the target, frame by frame, to transform the user target replacement received from the user into a modified user target replacement that acquires characteristics of the target in the corresponding original digital video file; and

inserting the modified user target replacement at a position of the target in each frame of the user selected altered digital video file in which the target appears in the original digital video file to substantially match at least the position, orientation and size of the selected target in the original digital video file to produce a personalized digital video file.

51. The process of claim 50, comprising:

sending the personalized digital video file to the user via the network.

52. The process of claim 50, comprising:

storing the personalized digital video file on a storage device; and

sending the storage device to the user.

53. The process of claim 50, comprising:

using the network to receive a digital file containing the user target replacement identified by the user.

54. The process of claim 50, comprising:

providing a digital library to store digital data of target replacements; and

providing access to the library to users on the network to allow each user to browse and select stored target replacements for making personalized digital video files,

wherein;

the user target replacement is retrieved from the library after the user selects the user target replacement.

55. The process of claim 54, wherein:

the library stores target replacements for products.

56. The process of claim 54, wherein:

the library stores target replacements for actors.

57. The process of claim 50, comprising:

triggering a logic trigger embedded in the personalized video when a specific image in the personalized video is displayed.

58. The process of claim 50, comprising:

providing a model library to store digital data of actor target replacements;

providing a product library to store digital data of product target replacements and providing access to the model library to users on the network to allow each user to browse and select stored actor target replacements for making personalized digital video files,

obtaining demographic information on either a user or a user selected actor target replacement;

selecting one or more product target replacements from the product library based on the obtained demographic information for insertion into the personalized digital video file requested by the user; and

inserting each user selected actor target replacement and each product target replacement into the user selected altered digital video file to produce a personalized digital video file.

59. The process of claim 50, wherein:

each altered digital video file is produced from a corresponding original digital video file by;

selecting a target in the original digital video file to be replaced by a target replacement, wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video file;

analyzing each frame of the original digital video file to track a change in the selected target in the original digital video file to capture data on the selected target, wherein the captured data includes at least information on a position, orientation and size of the selected target in the original digital video file; and

replacing the selected target with an image that resembles a continuation of a scene adjacent to the target in the original digital video file to produce the altered digital video file in which the selected target is removed.

60. The process of claim 50, comprising:

embedding a logic trigger associated with a specific image in the personalized video; and

triggering the logic trigger when the specific image in the personalized video is displayed.

85. The process of claim 50, wherein:

the user selected altered digital video file includes two or more selected targets that are replaced.

86. The process of claim 85, wherein:

a first selected target of the two or more selected targets is an actor in the original digital video file, and

a second selected target of the two or more selected targets is a non-actor object in the original digital video file.

87. The process of claim 50, comprising:

embedding a video identifier in the personalized digital video to allow for identification of the personalized digital video in a playback; and

embedding a target replacement identifier in the personalized digital video to allow for identification of the user target replacement in a playback.

88. The process of claim 87, comprising:

using the video identifier to track a number of times the personalized video is played; and

using the target replacement identifier to track a number of times the target replacement is playeddisplayed.

89. The process of claim 88, comprising:

embedding a logic trigger in the personalized video to trigger logic that counts a number of times that the user target replacement in the personalized video is playeddisplayed.

90. The process of claim 89, wherein:

the logic trigger includes a beginning logic trigger in a video frame where the user target replacement first appears in the personalized video and an ending trigger in a subsequent video frame where the user target replacement first disappears.

91. The process of claim 89, wherein:

the logic includes a counter connected on the Internet to receive a trigger signal generated by a playback of the personalized digital video with the embedded logic trigger.

92. The process of claim 89, wherein:

the logic includes a counter which is located in a device that plays the personalized video.

93. The process of claim 92, comprising:

transmitting values of the counter to the Internet when the device is connected to the Internet.

94. The process of claim 87, comprising:

storing information on the number of times the personalized video is played and the number of times the user target replacement is displayed in association with the video identifier for the personalized video and the target replacement identifier for the target replacement.

95. The process of claim 94, wherein:

the user target replacement is an image associated with a product or service.

61. A system for providing personalized digital video, comprising:

a network for communications;

a video library to store altered digital video files based on original digital video files, wherein at least one target in an original video file is removed in a corresponding altered digital video file and is substituted by an image that resembles a continuation of a scene adjacent to the target in a frame of the original digital video file, and wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video file; and

a computer in communication with the network and the video library, the computer operable to interface with a user via the network and to receive a request from the user for personalizing a user selected altered digital video file by replacing a target in a corresponding original digital video file with a user target replacement identified by the user,

wherein the computer is operable to

retrieve from the video library data on the target that is removed from the user selected altered digital video file, wherein the data includes at least information on a position, orientation and size of the target in the original digital video file for the user selected altered digital video file,

apply the retrieved data on the target, frame by frame, to transform the user target replacement received from the user into a modified user target replacement that acquires characteristics of the target in the corresponding original digital video file, and

insert the modified user target replacement at a position of the target in each frame of the user selected altered digital video file in which the target appears in the original digital video file to substantially match at least the position, orientation and size of the selected target in the original digital video file to produce a personalized digital video file for the user.

62. The system of claim 61, wherein:

the computer is operable to deliver the personalized digital video file to the user via the network.

63. The system of claim 61, wherein:

the network is the Internet.

64. The system of claim 61, wherein:

the network is a local area network.

65. The system of claim 61, wherein:

the system is operable to respond to a logic trigger embedded in the personalized video to trigger a counter that counts a number of times that the personalized digital video or a particular image in the personalized video is displayed.

66. The system of claim 65, wherein:

the counter is connected in the network to receive a trigger signal generated by a playback of the personalized digital video with the embedded logic trigger.

67. The system of claim 61, comprising:

a video processing computer in communication with the network to process a original digital video file to produce a corresponding altered digital video file stored in the video library,

the video processing computer operable to

select a target in original digital video to be replaced by a target replacement, wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video;

analyze each frame of the original digital video to track a change in the selected target in the original digital video to capture data on the selected target, wherein the captured data includes at least information on a position, orientation and size of the selected target in the original digital video;

replace the selected target with an image that resembles a continuation of a scene adjacent to the target in the original digital video to produce altered digital video in which the selected target is removed; and

store the captured data of the selected target and the altered digital video in an association with each other in the library for producing personalized digital video by the computer.

96. The system of claim 61, wherein:

a video identifier is embedded in the personalized digital video to allow for identification of the personalized digital video in a playback; and

a target replacement identifier is embedded in the personalized digital video to allow for identification of the user target replacement in a playback.

97. The system of claim 96, wherein:

the system is operable to

use the video identifier to track a number of times the personalized video is played; and

use the target replacement identifier to track a number of times the user target replacement is displayed.

98. The process of claim 97, wherein:

a logic trigger is embedded in the personalized video to trigger logic that counts a number of times that the user target replacement in the personalized video is displayed.

99. The system of claim 98, wherein:

the logic trigger includes a beginning logic trigger in a video frame where the user target replacement first appears in the personalized video and an ending trigger in a subsequent video frame where the user target replacement first disappears.

100. The system of claim 98, comprising:

a counter connected on the network to receive a trigger signal generated by a playback of the personalized digital video with the embedded logic trigger.

101. The system of claim 96, comprising:

a storage device connected to the network to store information on the number of times the personalized video is played and the number of times the user target replacement is displayed in association with the video identifier for the personalized video and the target replacement identifier for the target replacement.

102. The system of claim 101, wherein:

the storage device includes a lookup table to contain the video identifier for the personalized video, the target replacement identifier for the target replacement, the number of times the personalized video is played and the number of times the user target replacement is displayed in association with the video identifier for the personalized video and the target replacement identifier for the target replacement, and

wherein the system is operable to update the lookup table to track the playback of the personalized video and the user target replacement.

103. The system of claim 61, wherein:

an original digital video file includes a single frame image.

104. The system of claim 61, wherein:

an original digital video file includes a collage style video sequence of frames, wherein each frame is a composite of a plurality of still images.

105. The system of claim 61, wherein:

an original digital video file includes a video sequence of a plurality of still images.

68. A computer-implemented process for providing personalized video, comprising:

providing a user selected altered digital video file by replacing a target in a corresponding original digital video file with a user target replacement identified by the user;

obtaining data on the target that is removed from the user selected altered digital video file, wherein the data includes at least information on a position, orientation and size of the target in the original digital video file for the user selected altered digital video file;

applying the obtained data on the target, frame by frame, to transform the user target replacement into a modified user target replacement that acquires characteristics of the target in the corresponding original digital video file; and

inserting the modified user target replacement at a position of the target in each frame of the user selected altered digital video file in which the target appears in the original digital video file to substantially match at least the position, orientation and size of the selected target in the original digital video file to produce a personalized digital video file.

69. A storage medium having instructions stored thereon which, when executed by a processor, will cause the processor to perform operations for personalizing a user selected altered digital video file by replacing a target in a corresponding original digital video file with a user target replacement identified by the user, the operations comprising:

obtaining data on the target that is removed from the user selected altered digital video file, wherein the data includes at least information on a position, orientation and size of the target in the original digital video file for the user selected altered digital video file;

applying the obtained data on the target, frame by frame, to transform the user target replacement into a modified user target replacement that acquires characteristics of the target in the corresponding original digital video file; and

inserting the modified user target replacement at a position of the target in each frame of the user selected altered digital video file in which the target appears in the original digital video file to substantially match at least the position, orientation and size of the selected target in the original digital video file to produce a personalized digital video file.

70. A storage medium having instructions stored thereon which, when executed by a processor, will cause the processor to perform operations comprising:

selecting a target in original digital video to be replaced by a target replacement, wherein the target is a portion or an entirety of an actor or an object other than an actor in the original digital video;

analyzing each frame of the original digital video to track a change in the selected target in the original digital video to capture data on the selected target, wherein the captured data includes at least information on a position, orientation and size of the selected target in the original digital video; and

replacing the selected target with an image that resembles a continuation of a scene adjacent to the target in the original digital video to produce altered digital video in which the selected target is removed.