Interactive video manipulation

• US 6,954,498 B1
• Filed: 10/24/2000
• Issued: 10/11/2005
• Est. Priority Date: 10/24/2000
• Status: Expired due to Term

• Current Assignee: Avigilon Fortress Corporation (Motorola Solutions, Inc.)
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Lipton, Alan J.
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/694,712
• Time in Patent Office: 1,813 Days
• Field of Search: 375/240.08, 375/240.16, 375/240.26, 348/169, 348/416.1, 348699-700, 348/722, 348575-576, 348/578, 348/580, 348/584, 348/586, 348/589, 725/32, 725/36, 725/135, 345/418, 345/441, 345/473, 345/719, 345/722, 345/723, 345/753, 345/501
• US Class Current: 375/240.08
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

In a system for video processing, an object of interest is extracted from a video stream. The object from said video stream is analyzed and manipulated to obtain a synthetic character. A virtual video is assembled using the synthetic character.

Claims

1. A method comprising the steps of:

extracting an object of interest from a video stream;

analyzing said object from said video stream to obtain an analyzed object, said step of analyzing comprising at least one of the steps of determining a rigidity of said object, determining a periodic sequence corresponding to said object, or determining a periodic sequence corresponding to said object and at least one parameter describing an appearance of said object;

manipulating said analyzed object to obtain a synthetic character; and

assembling a virtual video using said synthetic character.

2. The method of claim 1, wherein said step of extracting comprises the step of employing a stochastic background modeling technique to detect said object in said video stream.

3. The method of claim 1, wherein said step of extracting comprises the step of employing a motion detection technique to detect said object in said video stream.

4. The method of claim 1, wherein said step of extracting comprises the step of employing a segmentation technique to detect said object in said video stream.

5. The method of claim 1, wherein said step of extracting comprises the step of extracting background components from said video stream.

6. The method of claim 1, wherein said step of extracting comprises the steps of:

determining if a pixel in a frame of said video stream represents an object that is moving based on a stochastic model of a background scene in said frame; and

clustering pixels in said frame that represent said object that is moving.

7. The method of claim 1, further comprising the step of tracking said object.

8. The method of claim 7, wherein said step of tracking comprises the steps of:

identifying a position of said object in a frame of said video stream;

identifying candidate objects in a next frame of said video stream; and

comparing said object in said frame with candidate objects in said next frame to determine a next position of said object in said next frame.

9. The method of claim 1, wherein said step of determining said rigidity comprises the step of determining a residual flow for said object.

10. The method of claim 1, wherein said periodic sequence represents one cycle of motion of said object over a series of frames, and wherein said periodic sequence comprises, for each frame of said set of frames, a visual appearance of said object and a frame-to-frame displacement of said object.

11. The method of claim 1, wherein said step of determining said periodic sequence comprises the steps of:

collecting a set of visual templates of said object from a series of frames of said video stream, said set of visual templates comprising at least one complete period of motion of said object; and

matching a present visual template of said object with each visual template of said set of visual templates to determine a starting point of said period of motion of said object.

12. The method of claim 11, wherein said step of matching comprises the steps of:

determining a convolution of said present visual template with each visual template of said set of visual templates; and

selecting said starting point of said period of motion based on a minimum of said convolution.

13. The method of claim 1, further comprising the step of transmitting or storing said periodic sequence and said at least one parameter.

14. The method of claim 1, wherein said step of manipulating is directed by a user.

15. The method of claim 1, wherein said step of manipulating is directed by a computation engine.

16. The method of claim 1, wherein said synthetic character is a hybrid based on said object of interest and computer-generated imagery.

17. The method of claim 1, further comprising the step of providing a second synthetic character generated by a computer graphics engine, and wherein said step of assembling comprises the step of assembling said virtual video using said synthetic character and said second synthetic character.

18. The method of claim 1, wherein said step of assembling comprises the step of inserting said synthetic character into said video stream.

19. The method of claim 18, wherein said synthetic character is inserted based on rigidity and periodicity of said synthetic character.

20. The method of claim 18, wherein said step of inserting comprises the steps of:

selecting a starting frame in said video stream and a position within said starting frame for inserting said synthetic character; and

inserting a periodic sequence corresponding to said synthetic character beginning in said starting frame and at said position.

21. The method of claim 20, wherein said periodic sequence represents one cycle of motion of said synthetic character, and wherein said periodic sequence comprises, for each frame of said set of frames, a visual template of said synthetic character and a frame-to-frame displacement of said synthetic character.

22. The method of claim 21, wherein said step of inserting further comprises the step of multiplying each visual template by a scale factor to adjust a size of said synthetic character.

23. The method of claim 21, wherein said step of inserting further comprises the step of multiplying each frame-to-frame displacement by a time factor to adjust a speed of motion of said synthetic character.

24. The method of claim 21, wherein said step inserting further comprises the step of applying a flip operator to each visual template to reverse a direction of motion of said synthetic character relative to a direction of motion of said synthetic character in said periodic sequence.

25. The method of claim 18, wherein said synthetic character is occluded in said video stream.

26. The method of claim 18, wherein said step of inserting comprises the step of modifying at least one of appearance, scale, position, speed, direction of motion, and timing, prior to insertion of said synthetic character into said video stream.

27. The method of claim 1, wherein said step of assembling comprises the step of removing said synthetic character from said video stream.

28. The method of claim 27, wherein said step of removing comprises the step of filling in pixels in frames of said video stream with pixels from a background model for said synthetic character removed from said video stream.

29. The method of claim 27, wherein said step of removing comprises the step of repairing at least one of an uncovered background, a foreground object, and another synthetic character.

30. The method of claim 1, further comprising the step of determining functional areas within said video stream.

31. The method of claim 1, further comprising performing the method of claim 1 for a plurality of objects of interest in said video stream.

32. The method of claim 1, wherein said steps of extracting, analyzing, manipulating, and assembling are performed in real time.

33. The method of claim 1, wherein at least one of said steps of extracting, analyzing, manipulating, and assembling is performed in non-real time.

34. The method of claim 1, wherein said video stream comprises a background for a game.

35. The method of claim 1, wherein said video stream comprises a simulation.

36. The method of claim 1, wherein said video stream comprises a teleconference.

37. The method of claim 1, wherein said video stream comprises a distance education presentation.

38. A computer system to perform the method of claim 1.

39. A system comprising means for processing to perform the method of claim 1 and means for rendering said virtual video.

40. A computer-readable medium comprising software to perform the method of claim 1.

41. A method comprising the steps of:

obtaining a video stream as a setting for one of a video game, a simulation, a teleconference, and a distance education presentation;

tracking a moving object in said video stream;

analyzing said moving object to obtain an analyzed moving object, said analyzing step comprising representing said moving object by a periodic sequence;

generating a synthetic character based on a said analyzed moving object; and

assembling a virtual video based on said synthetic character and said video stream.

42. The method of claim 41, wherein said step of generating is in response to a user of one of said video game, said simulation, said teleconference, and said distance education presentation.

43. The method of claim 41, wherein said step of assembling comprises the step of inserting said moving object into said virtual video based on said synthetic character.

44. The method of claim 41, wherein said step of assembling comprises the step of removing said moving object from said virtual video based on said synthetic character.

45. A computer system to perform the method of claim 41.

46. A system comprising means for processing to perform the method of claim 41 and means for rendering said virtual video.

47. A computer-readable medium comprising software to perform the method of claim 41.

48. A method comprising the steps of:

extracting in real time a background model from a video stream;

generating in real time a synthetic character;

identifying a functional area in said video stream; and

assembling in real time a virtual video based on said background model, said synthetic character, and said functional area.

49. A method as in claim 48, wherein said step of generating comprises generating said synthetic character using a computer graphics engine.

50. A method as in claim 48, further comprising the step of extracting in real time an object of interest from said video stream, and wherein said step of generating comprises generating said synthetic character using said object.

51. A method as in claim 48, further comprising the step of extracting in real time an object of interest from said video stream, and wherein said step of generating comprises generating said synthetic character using said object and a computer graphics engine.

52. A computer system to perform the method of claim 48.

53. A system comprising means for processing to perform the method of claim 48 and means for rendering said virtual video.

54. A computer-readable medium comprising software to perform the method of claim 48.