Insertion of virtual video into live video

• US 8,310,521 B2
• Filed: 04/30/2007
• Issued: 11/13/2012
• Est. Priority Date: 04/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Bobick, Aaron
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US11/742,504
• Publication Number:
  US 20080267578A1
• Time in Patent Office: 2,024 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-356, 709201-207, 709217-248
• US Class Current: 348/14.15
• CPC Class Codes: G11B 27/036 Insert-editing

Abstract

The present virtual video muting technique seamlessly inserts a virtual video into a live video when the user does not want to reveal his/her actual activity. The virtual video is generated based on real video frames captured earlier, and thus makes the virtual video appear to be real.

Claims

1. A computer-implemented process inserting virtual video into a live video, comprising using a computer to perform the process actions of:

inputting video and associated audio of a meeting participant;

recording portions of the input video as video clips in a video clip database;

outputting the input video and audio if muting is not desired; and

if muting is desired,

taking a frame from the input video just prior to transitioning to mute and selecting a recorded frame in the video clip database similar to the input video frame prior to transitioning to mute as a transition point to transition to virtual video, and

generating virtual video using the recorded portions of the input video from the video clip database starting from the selected recorded frame until muting is disabled, comprising;

playing a video clip from the video clip database;

if the playing video clip meets a prescribed transition point, taking a frame from the playing video clip and finding a similar frame in the currently playing or another video clip in the video clip database; and

playing the video clip starting at the similar frame.

2. The computer-implemented process of claim 1 further comprising the process actions of:

encoding the virtual video or the input video; and

transmitting the virtual video or the input video over the network to one or more other meeting participants.

3. The computer-implemented process of claim 1 further comprising the process action of finding the best point to transition to the virtual video from the input video.

4. The computer-implemented process of claim 3 wherein finding the best point to transition to the virtual video from the input video comprises:

taking a video frame from the input video just prior to muting and searching for a recorded frame in the video clip database similar to the video frame from the input video just prior to muting as a transition point to play the recorded video clip corresponding to the similar frame, starting from the similar recorded frame.

5. The computer-implemented process of claim 1 wherein similarity between frames is determined by comparing pixel colors in corresponding locations between frames.

6. The computer-implemented process of claim 1 wherein the prescribed transition point is determined by employing a random number generator to determine when to transition to another video clip.

7. The computer-implemented process of claim 1 wherein the prescribed transition point is determined by a prescribed transition pattern that is inversely proportional to the number of possible future transitions to other video clips.

8. The computer-implemented process of claim 1 wherein the prescribed transition point is determined by a predetermined transition pattern.

9. The computer-implemented process of claim 3 further comprising using an entry mirror which blends the input video and the virtual video thereby aiding the meeting participant in positioning themselves at the transition to the virtual video from the input video.

10. The computer-implemented process of claim 3 further comprising using an exit mirror which blends the virtual video and the input video thereby aiding the meeting participant in positioning themselves at the transition to the input video from the virtual video.

11. The computer-implemented process of claim 1 wherein text transcription of the audio of other meeting participants takes place while audio for the input video is muted.

12. The computer-implemented process of claim 1 wherein instant replay functionality of audio or video is available after the audio for the input video has been muted and un-muted.

13. A computer-readable medium having computer-executable instructions for performing the computer-implemented process recited in claim 1.

14. A system for generating virtual video, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

input video of a video conference meeting participant;

save clips of the input video into a video clip database;

detect whether muting is requested;

select a frame in the video clip database similar to a frame in the input video just prior to detecting the mute request as a transition point;

if muting is requested, starting from the transition point using saved clips of the input video to create virtual video of the video conference meeting participant comprising;

playing a video clip from the video clip database;

if the playing video clip meets a prescribed transition point, taking a frame from the playing video clip and finding a similar frame in the currently playing or another video clip in the video clip database; and

playing the video clip starting at the similar frame.

15. The system of claim 14 wherein the virtual video is transmitted to other video conference meeting participants instead of the input video of the video conference meeting participant.

16. A computer-implemented process for creating virtual video, comprising:

(a) inputting video and associated audio of a meeting participant;

(b) storing portions of the input video in a video clip data base;

(c) if muting is requested, taking a frame just prior to the muting and searching for a in a video clip in the video clip data base that is similar;

(d) playing the video clip in the video clip database, starting from the frame in the video clip that is similar to the frame just prior to the muting until criteria for transition to another video clip is met;

(e) if criteria for transition to another video clip is met, taking a frame from the currently playing video clip and selecting a similar frame in a different video clip;

(f) playing the different video clip from the selected frame until an unmute command is received or criteria for transitioning to another video clip are met;

(g) if criteria for transition to another video clip is met, repeating (e) and (f);

(h) if an unmute command is received, playing the input video.

17. The computer-implemented process of claim 16 wherein a frame is determined to be similar to another frame by comparing the color of pixels in corresponding locations.

18. The computer-implemented process of claim 16 wherein transitions between video clips are conducted based on a predetermined pattern.

19. The computer-implemented process of claim 16 wherein a mute indicator is displayed when associated audio for the input video is muted.