Scheme for interactive video manipulation and display of moving object on background image

• US 6,215,505 B1
• Filed: 06/18/1998
• Issued: 04/10/2001
• Est. Priority Date: 06/20/1997
• Status: Expired due to Term

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Satou, Takashi, Akutsu, Akihito, Tonomura, Yoshinobu, Taniguchi, Yukinobu, Minami, Kenichi
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Sealey, Lance W.
• Application Number:
  US09/099,189
• Time in Patent Office: 1,027 Days
• Field of Search: 345/473, 345/328, 345/439, 345/435, 386/46
• US Class Current: 345/473
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 13/00 Animation

Abstract

A scheme for interactive video manipulation and display of a moving object on a background image is disclosed. In this scheme a background image stored in a background image storage unit is read out and displayed on a screen, and a manipulation target spatial position is set on the screen. Then, a partial image to be synthesized and displayed is uniquely specified from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the setting step, and the partial image as specified by the specifying step is read out from a partial image storage unit, and synthesized and displayed at the manipulation target spatial position on the background image. A scheme for realizing a variable speed video playback with a playback speed different from a normal one is also disclosed.

Claims

1. An apparatus for interactive video manipulation and display, comprising:

a background image storage unit for storing a background image;

a display unit for displaying the background image storage unit stored in the background image storage unit;

a partial image storage unit for storing partial images which are set in correspondence to spatial position on the background image and representing an object moving on the background image;

a setting unit for setting a manipulation target spatial position on a screen of the display unit;

a specification unit for uniquely specifying a partial image to be synthesized and displayed according to the manipulation target spatial position set by the setting unit wherein the specifying unit traces a line which continuously maps spatial and time positions, established by the setting unit, of successive partial images; and

a synthesis display unit for reading out the partial image as specified by the specifying unit from the partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image displayed by the display unit.

2. The apparatus of claim 1, wherein the setting unit is provided in a form of a display integral touch screen.

3. The apparatus of claim 1, further comprising a unit for producing the background image to be stored in the background image storage unit from an original video containing consecutive frame images.

4. The apparatus of claim 1, further comprising a unit for extracting the partial images to be stored in the partial image storage unit from an original video from which the background image is produced.

5. The apparatus of claim 1, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the specifying unit converts sections of the mapping which are close to each other in time into one-to-one mapping.

6. The apparatus of claim 1, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the specifying unit selects one value from multiple values according to a prescribed rule.

7. The apparatus of claim 1, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, if the setting unit sets successive manipulation target spatial positions continuously, the specifying unit maps the successive manipulation target spatial positions into corresponding continuous time positions, and if the setting unit sets successive manipulation target spatial positions discontinuously, the specifying unit selects one value from multiple values according to a prescribed rule.

8. The apparatus of claim 1, wherein the synthesis display unit synthesizes and displays the partial image and the background image by enlarging or contracting the partial image and the background image, when an enlarging or contracting mode is specified.

9. The apparatus of claim 1, wherein the synthesis display unit synthesizes and displays successive partial images continuously or at constant time interval for a prescribed period of time starting from or ending at a time position corresponding to the manipulation target spatial position set by the setting unit, when an arbitrary position playback mode setting and a playback direction setting are made.

10. The apparatus of claim 1, wherein the synthesis display unit erases the partial image that is synthesized and displayed once after a prescribed period of time.

11. The apparatus of claim 1, further comprising a sound playback unit for sequentially obtaining time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the setting unit, obtaining sound data corresponding to a period between a current time position and an immediately previous time position, and playbacking the sound data within the prescribed time interval.

12. The apparatus of claim 11, wherein the sound playback unit playbacks the sound data by compressing the sound data when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.

13. The apparatus of claim 1, wherein the setting unit is provided in a form of a pointing device for moving a cursor displayed by the display unit, and the display unit displays the cursor in a transparent form showing only a contour at a time of setting the manipulation target spatial position.

14. A method for interactive video manipulation and display, comprising the steps of:

reading out a background image stored in a background image storage unit and displaying the background image on a screen;

setting a manipulation target spatial position on the screen;

uniquely specifying a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the setting step, wherein the specifying step traces a line which continuously maps spatial and time positions, established by the setting step, of successive partial images; and

reading out the partial image as specified by the specifying step from a partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image.

15. The method of claim 14, further comprising the step of producing the background image to be stored in the background image storage unit from an original video containing consecutive frame images.

16. The method of claim 14, further comprising the step of extracting the partial images to be stored in the partial image storage unit from an original video from which the background image is produced.

17. The method of claim 14, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the specifying step converts sections of the mapping which are close to each other in time into one-to-one mapping.

18. The method of claim 14, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the specifying step selects one value from multiple values according to a prescribed rule.

19. The method of claim 14, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, if the setting step sets successive manipulation target spatial positions continuously, the specifying step maps the successive manipulation target spatial positions into corresponding continuous time positions, and if the setting step sets successive manipulation target spatial positions discontinuously, the specifying step selects one value from multiple values according to a prescribed rule.

20. The method of claim 14, wherein the synthesizing and displaying step synthesizes and displays the partial image and the background image by enlarging or contracting the partial image and the background image, when an enlarging or contracting mode is specified.

21. The method of claim 14, wherein the synthesizing and displaying step synthesizes and displays successive partial images continuously or at constant time interval for a prescribed period of time starting from or ending at a time position corresponding to the manipulation target spatial position set by the setting step, when an arbitrary position playback mode setting and a playback direction setting are made.

22. The method of claim 14, wherein the synthesizing and displaying step erases the partial image that is synthesized and displayed once after a prescribed period of time.

23. The method of claim 14, further comprising the step of sequentially obtaining time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the setting step, obtaining sound data corresponding to a period between a current time position and an immediately previous time position, and playbacking the sound data within the prescribed time interval.

24. The method of claim 23, wherein the playbacking step playbacks the sound data by compressing the sound data when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.

25. The method of claim 14, wherein the setting step sets the manipulation target spatial position using a pointing device for moving a cursor displayed on the screen, which is displayed in a transparent form showing only a contour at a time of setting the manipulation target spatial position.

26. An article of manufacture, comprising:

a computer usable medium having computer readable program code means embodied therein for causing a computer to function as an apparatus for interactive video manipulation and display, the computer readable program code means includes;

first computer readable program code means for causing said computer to read out a background image stored in a background image storage unit and display the background image on a screen;

second computer readable program code means for causing said computer to set a manipulation target spatial position on the screen;

third computer readable program code means for causing said computer to uniquely specify a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the second computer readable program code means, wherein the third computer readable program code means traces a line which continuously maps the spatial and time positions, established by the second computer readable program code means, of successive partial images; and

fourth computer readable program code means for causing said computer to read out the partial image as specified by the third computer readable program code means from a partial image storage unit, and synthesizing displaying the partial image at the manipulation target spatial position on the background image.

27. The article of manufacture of claim 26, further comprising fifth computer readable program code means for causing said computer to produce the background image to be stored in the background image storage unit from an original video containing consecutive frame images.

28. The article of manufacture of claim 26, further comprising fifth computer readable program code means for causing said computer to extract the partial images to be stored in the partial image storage unit from an original video from which the background image is produced.

29. The article of manufacture of claim 26, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the third computer readable program code means converts sections of the mapping which are close to each other in time into one-to-one mapping.

30. The article of manufacture of claim 26, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, the third computer readable program code means selects one value from multiple values according to a prescribed rule.

31. The article of manufacture of claim 26, wherein when the mapping from the manipulation target spatial position to the time position becomes multi-valued, if the setting step sets successive manipulation target spatial positions continuously, the specifying step maps the successive manipulation target spatial positions into corresponding continuous time positions, and if the setting step sets successive manipulation target spatial positions discontinuously, the specifying step selects one value from multiple values according to a prescribed rule.

32. The article of manufacture of claim 26, wherein the fourth computer readable program code means synthesizes and displays the partial image and the background image by enlarging or contracting the partial image and the background image, when an enlarging or contracting mode is specified.

33. The article of manufacture of claim 26, wherein the fourth computer readable program code means synthesizes and displays successive partial images continuously or at constant time interval for a prescribed period of time starting from or ending at a time position corresponding to the manipulation target spatial position set by the second computer readable program code means, when an arbitrary position playback mode setting and a playback direction setting are made.

34. The article of manufacture of claim 26, wherein the fourth computer readable program code means erases the partial image that is synthesized and displayed once after a prescribed period of time.

35. The article of manufacture of claim 26, further comprising the fifth computer readable program code means for causing said computer to sequentially obtain time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the second computer readable program code means, obtain sound data corresponding to a period between a current time position and an immediately previous time position, and playback the sound data within the prescribed time interval.

36. The article of manufacture of claim 35, wherein the fifth computer readable program code means causes playbacks the sound data by compressing the sound data when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.

37. The article of manufacture of claim 26, wherein the second computer readable program code means sets the manipulation target spatial position using a pointing device for moving a cursor displayed on the screen, which is displayed in a transparent form showing only a contour at a time of setting the manipulation target spatial position.

38. An apparatus for interactive video manipulation and display, comprising:

a background image storage unit for storing a background image;

a display unit for displaying the background image stored in the background image storage unit;

a partial image storage unit for storing partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image;

a setting unit for setting a manipulation target spatial position on a screen of the display unit;

a specifying unit for uniquely specifying a partial image to be synthesized and displayed according to the manipulation target spatial position set by the setting unit; and

a synthesis display unit for reading out the partial image as specified by the specifying unit from the partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image displayed by the display unit, by synthesizing an displaying the partial image and the background image by enlarging or contracting the partial image and the background image when an enlarging or contracting mode is specified.

39. An apparatus for interactive video manipulation and display, comprising:

a background image storage unit for storing a background image;

a display unit for displaying the background image stored in the background image storage unit;

a partial image storage unit for storing partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image;

a setting unit for setting a manipulation target spatial position on a screen of the display unit;

a specifying unit for uniquely specifying a partial image to be synthesized and displayed according to the manipulation target spatial position set by the setting unit;

a synthesis display unit for reading out the partial image as specified by the specifying unit from the partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image displayed by the displayed unit; and

a sound playback unit for sequentially obtaining time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the setting unit, obtaining sound data corresponding to a period between a current time position and an immediately previous time position, and playbacking the sound data within the prescribed time interval by compressing the sound data when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.

40. A method for interactive video manipulation and display, comprising the steps of:

reading out a background image stored in a background image storage unit and displaying the background image on a screen;

setting a manipulation target spatial position on the screen;

uniquely specifying a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the setting step; and

reading out the partial image as specified by the specifying step from a partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image, by synthesizing and displaying the partial image and the background image by enlarging or contracting mode is specified.

41. A method for interactive video manipulation and display, comprising the steps of:

reading our a background image stored in the background image storage unit and displaying the background image on a screen;

setting a manipulation target spatial position on the screen;

uniquely specifying a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the setting step;

reading out the partial image as specified by the specifying step from a partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image; and

sequentially obtaining time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the setting step, obtaining sound data corresponding to a period between a current time position and an immediately previous time position, and playbacking the sound data within the prescribed time interval by compressing the sound data when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.

42. An article of manufacture, comprising:

a computer usable medium having computer readable program code means embodied therein for causing a computer to function as an apparatus for interactive video manipulation and display, the computer readable program code means includes;

first computer readable program code means for causing said computer to read out a background image stored in a background image storage unit and display the background image on the screen;

second computer readable program code means for causing said computer to set a manipulation target spatial position on the screen;

third computer readable program code means for causing said computer to uniquely specify a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the second computer readable program code means; and

fourth computer readable program code means for causing said computer to read out the partial image as specified by the third computer readable program code means from a partial image storage unit, and synthesize and display the partial image at the manipulation target spatial position on the background image by synthesizing and displaying the partial image and the background image by enlarging or contracting the partial image and the background image when an enlarging or contracting mode is specified.

43. An article of manufacture, comprising:

a computer usable medium having computer readable program code means embodied therein for causing a computer to function as an apparatus for interactive video manipulation and display, the computer readable program code means includes;

first computer readable program code means for causing said computer to read out a background image stored in a background image storage unit and display the background image on a screen;

second computer readable program code means for causing said computer to set a manipulation target spatial position on the screen;

third computer readable program code means for causing said computer to uniquely specify a partial image to be synthesized and displayed from partial images which are set in correspondence to spatial positions on the background image and representing an object moving on the background image, according to the manipulation target spatial position set by the second computer readable program code means;

fourth computer readable program code means for causing said computer to read out the partial image as specified by the third computer readable program code means from a partial image storage unit, and synthesizing and displaying the partial image at the manipulation target spatial position on the background image; and

fifth computer readable program code means for causing said computer to sequentially obtain time positions at which the partial image to be synthesized and displayed exists sequentially at a prescribed time interval according to manipulation target spatial positions sequentially set by the second computer readable program code means, obtain sound data corresponding to a period between a current time position and an immediately previous time position, and playback the sound data within the prescribed time interval.

44. The article of manufacture of claim 43, wherein the fifth computer readable program code means causes playbacks the sound data by compressing the sound when a difference between the current time position and the immediately previous time position is longer than the prescribed time interval or by expanding the sound data when the difference is shorter than the prescribed time interval, so that a playback of the sound data is finished within the prescribed time interval.