Method of spatio-temporally integrating/managing a plurality of videos and system for embodying the same, and recording medium for recording a program for the method

• US 6,411,339 B1
• Filed: 06/04/1998
• Issued: 06/25/2002
• Est. Priority Date: 10/04/1996
• Status: Expired due to Fees

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Hamada, Hiroshi, Akutsu, Akihito, Tonomura, Yoshinobu
• Primary Examiner(s): Garber, Wendy R.
• Assistant Examiner(s): Villecco, John M
• Application Number:
  US09/090,642
• Time in Patent Office: 1,482 Days
• Field of Search: 348/208, 348/36, 348/218, 348/584, 348/586, 348/39, 348/159, 348/157, 348/699, 348/700, 382/103, 382/190, 382/195, 382/284, 382/294
• US Class Current: 348/584
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

A camcorder operation condition detecting portion (103) reads image data string and detects camcorder on/off information and camcorder operating information. A video splitting portion (104) splits videos into respective shots based on the camcorder on/off information. A subject/background separating portion (105) separates a subject and a background based on the camcorder operating information, etc. A subject motion information extracting portion (106) correlates subject information separated every frame between frames. A video taken space resynthesizing portion (107) resynthesizes video taken spaces based on the camcorder operating information and the background. A shot-to-shot relation calculating portion (108) calculates spatial shot-to-shot relations between a plurality of video taken spaces. Hence, respective information of camcorder on/off, camcorder operation, subject, subject motion, resynthesized background, and shot-to-shot relation information can be spatio-temporally managed/stored, and the video taken spaces and one or more subjects can be resynthesized, displayed, or output in accordance with user'"'"'s requests, etc.

Claims

1. A system for spatio-temporally integrating/managing a plurality of videos, comprising:

an image data string memory portion for reading video data to save video data as data string;

a camcorder operation condition detecting portion for reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information;

a video splitting portion for splitting videos of the data string into respective shots based on the camcorder on/off information;

a subject/background separating portion for separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties;

a subject motion information extracting portion for correlating subject information separated every frame between frames;

a video taken space resynthesizing portion for resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame;

a shot-to-shot relation calculating portion for calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing portion respectively, based on a plurality of shots being separated; and

a video structure information managing/storing portion for managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.

2. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, further comprising:

a video structure information transmitter/receiver for transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.

3. A system for spatio-temporally integrating/managing a plurality of videos according to claim 2, further comprising:

a resynthesizer for resynthesizing one or plural video taken spaces and one or plural subjects based on information stored/managed in the video structure information managing/storing portion in compliance with one or both of predetermined conditions and user'"'"'s requests;

a display unit for displaying resynthesized videos in the resynthesizer;

a user input portion for inputting the user'"'"'s requests for resynthesis based on the videos which are displayed on the display unit; and

an output portion for outputting the videos which are displayed on the display unit to an external device in a digital or analogue manner.

4. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, wherein the camcorder operation condition detecting portion comprises,

a linear component calculating portion for calculating respective horizontal/vertical linear components of image frames read from the image data string memory portion,

an image data rearrangement portion for rearranging spatio-temporal placements which are occupied by the images including calculated respective horizontal/vertical linear components,

a video information filter processing portion for applying filter process to the image data derived by rearrangement,

a feature extracting portion for extracting features from results of the filter process, and

a statistical feature analyzing portion for detecting the camcorder on/off information and the camcorder operation information by analyzing statistically the extracted features.

5. A system for spatio-temporally integrating/managing a plurality of videos according to claim 4, wherein the image data rearrangement portion comprises an image data string rearrangement means for rearranging image data string into plural sheets of spatio-temporal sectional images which include normals of the image and a time axis.

6. A system for spatio-temporally integrating/managing a plurality of videos according to claim 4, wherein the video information filter portion comprises a segment detecting means for detecting edges or lines of the video information of rearranged image data.

7. A system for spatio-temporally integrating/managing a plurality of videos according to claim 6, wherein the feature extracting portion comprises an integration means for adding information as for detected edges or lines along a normal direction of the images.

8. A system for spatio-temporally integrating/managing a plurality of videos according to claim 4, wherein the statistical feature analyzing portion comprises a camcorder on/off detecting means for detecting straight lines of extracted features, which are represented as two-dimensional images having the time axis and the space axis, perpendicular to the time axis, to calculate the camcorder on/off information.

9. A system for spatio-temporally integrating/managing a plurality of videos according to claim 4, wherein the statistical feature analyzing portion comprises,

a correlating means for correlating space coordinates by comparing spatial distributions at any two moments of extracted features which are represented on two-dimensional images having the time axis and the space axis, and

a camcorder operation parameter calculating means for calculating camcorder operation parameters by processing statistically correlated space coordinates.

10. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, wherein the subject/background separating portion comprises,

a camcorder operation canceling portion for canceling the camcorder operation between adjacent image data based on the camcorder operation information detected by the camcorder operation condition detector,

an image data comparator for comparing the image data from which the camcorder operation information are canceled,

a comparison image data comparator for comparing adjacent comparison image data, and

an area extracting portion for extracting an subject area from comparison data which are calculated by the comparison image data comparator.

11. A system for spatio-temporally integrating/managing a plurality of videos according to claim 10, wherein the camcorder operation canceling portion comprises a video frame deforming means for deforming/displacing adjacent image frames based on the camcorder operation information to cancel change/displacement caused by the camcorder operation.

12. A system for spatio-temporally integrating/managing a plurality of videos according to claim 10, wherein the image data comparator comprises a difference processing means for executing difference process of brightness and color information between adjacent image data from which the camcorder operations are canceled.

13. A system for spatio-temporally integrating/managing a plurality of videos according to claim 10, wherein the area extracting portion comprises,

a binarization processing means for carrying out binarization process of the comparison data,

a labeling means for carrying out labeling binarization data which have been subjected to the binarization process,

a physical feature calculating means for calculating physical features in a labeled area, and

a collating means for collating calculated physical features with predetermined conditions to extract the subject area.

14. A system for spatio-temporally integrating/managing a plurality of videos according to claim 13, wherein the subject/background separating portion comprises a background extracting means for extracting the background by subtracting the subject area per frame extracted by the area extracting portion from the frame image.

15. A system for spatio-temporally integrating/managing a plurality of videos according to claim 13, wherein the subject motion information extracting portion comprises a collating means for collating physical features of adjacent areas in time of the subject areas extracted frame by frame by the area extracting portion with predetermined conditions to correspond the subject information between the frames.

16. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, wherein the video taken space resynthesizing portion comprises a space superposing means for superposing adjacent image frames into a continuous video taken space by deforming/displacing the adjacent image frames based on the camcorder operation information detected by the camcorder operation condition detecting portion.

17. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, wherein the shot-to-shot relation calculating portion comprises a video taken space transforming means for transforming a video taken space per shot generated by the video taken space resynthesizing portion so as to make size and positions thereof equal between video taken spaces.

18. A system for spatio-temporally integrating/managing a plurality of videos according to claim 1, wherein the video structure information managing/storing portion comprises a means for spatio-temporally managing/storing by data-compressing extracted subject information, the camcorder operation condition information, the background information, and the shot-to-shot relation information, video data, and all or a part of video taken spaces of a plurality of videos which are resynthesized by using the camcorder operation condition information, the background information, and the shot-to-shot relation information, the video data.

19. A system for spatio-temporally integrating/managing a plurality of videos according to claim 18, wherein the video structure information managing/storing portion comprises,

a means for data-compressing the extracted subject information as the still images by developing the extracted subject information into the space, and

a means for data-compressing, as the still images, video taken spaces of a plurality of videos which are resynthesized.

20. A method of spatio-temporally integrating/managing a plurality of videos, comprising:

an image data string saving step of reading video data to save the video data as data string in an image data string memory portion;

a camcorder operation condition detecting step of reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information;

a video splitting step of splitting videos of the data string into respective shots based on the camcorder on/off information;

a subject/background separating step of separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties;

a subject motion information extracting step of correlating subject information separated every frame between frames;

a video taken space resynthesizing step of resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame;

a shot-to-shot relation calculating step of calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing step respectively, based on a plurality of shots being separated; and

a video structure information managing/storing step of managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.

21. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, further comprising:

a video structure information transmitting/receiving step of transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.

22. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, further comprising, after the video structure information managing/storing step:

a resynthesizing step of resynthesizing one or plural video taken spaces and one or plural subjects based on stored/managed information in compliance with one or both of predetermined conditions and user'"'"'s requests; and

a display or outputting of displaying or outputting videos resynthesized by the resynthesizing step.

23. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, wherein the camcorder operation condition detecting step comprises,

a linear component calculating step of calculating respective horizontal/vertical linear components of image frames read by the image data string memory step,

an image data rearrangement step of rearranging spatio-temporal placements which are occupied by the images including calculated respective horizontal/vertical linear components,

a video information filter processing of applying filter process to the image data derived by rearrangement,

a feature extracting step of extracting features from results of the filter process, and

a statistical feature analyzing step of detecting the camcorder on/off information and the camcorder operation information by analyzing statistically the extracted features.

24. A method of spatio-temporally integrating/managing a plurality of videos according to claim 23, wherein the image data rearrangement step rearranges image data string into plural sheets of spatio-temporal sectional images which include normals of the image and a time axis.

25. A method of spatio-temporally integrating/managing a plurality of videos according to claim 23, wherein the video information filter step detects edges or lines of the video information of image data obtained by rearrangement.

26. A method of spatio-temporally integrating/managing a plurality of videos according to claim 25, wherein the feature extracting step extracts features by adding information as for detected edges or lines along a normal direction of the images.

27. A method of spatio-temporally integrating/managing a plurality of videos according to claim 23, wherein the statistical feature analyzing step detects straight lines of extracted features, which are represented as two-dimensional images having the time axis and the space axis, perpendicular to the time axis, to calculate the camcorder on/off information.

28. A method of spatio-temporally integrating/managing a plurality of videos according to claim 23, wherein the statistical feature analyzing step comprises,

a correlating step of correlating space coordinates by comparing spatial distributions at any two moments of extracted features which are represented on two-dimensional images having the time axis and the space axis, and

a camcorder operation parameter calculating step of calculating camcorder operation parameters by processing statistically correlated space coordinates.

29. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, wherein the subject/background separating step comprises,

a camcorder operation canceling step of canceling the camcorder operation between adjacent image data based on the camcorder operation information detected by the camcorder operation condition detecting step,

an image data comparing step of comparing the image data from which the camcorder operation information are canceled,

a comparison image data comparing step of comparing adjacent comparison image data, and

an area extracting step of extracting an subject area from comparison data which are calculated by the comparison image data comparing step.

30. A method of spatio-temporally integrating/managing a plurality of videos according to claim 29, wherein the camcorder operation canceling step deforms/displaces adjacent image frames based on the camcorder operation information to cancel change/displacement caused by the camcorder operation.

31. A method of spatio-temporally integrating/managing a plurality of videos according to claim 29, wherein the image data comparing step executes difference process of brightness and color information between adjacent image data from which the camcorder operations are canceled.

32. A method of spatio-temporally integrating/managing a plurality of videos according to claim 29, wherein the area extracting step comprises,

a binarization processing step of carrying out binarization process of the comparison data,

a labeling step of carrying out labeling binarization data which have been subjected to the binarization process,

a physical feature calculating step of calculating physical features in a labeled area, and

a collating step of collating calculated physical features with predetermined conditions to extract the subject area.

33. A method of spatio-temporally integrating/managing a plurality of videos according to claim 32, wherein the subject/background separating step extracts the background by subtracting the subject area per frame extracted by the area extracting step from the frame image.

34. A method of spatio-temporally integrating/managing a plurality of videos according to claim 32, wherein the subject motion information extracting step collates physical features of adjacent areas in time of the subject areas extracted frame by frame by the area extracting step with predetermined conditions to correlate the subject information between the frames.

35. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, wherein the video taken space resynthesizing step superposes adjacent image frames into a continuous video taken space by deforming/displacing the adjacent image frames based on the camcorder operation information detected by the camcorder operation condition detecting portion.

36. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, wherein the shot-to-shot relation calculating step transforms a video taken space per shot generated by the video taken space resynthesizing portion so as to make size and positions thereof equal between video taken spaces.

37. A method of spatio-temporally integrating/managing a plurality of videos according to claim 20, wherein the video structure information managing/storing step spatio-temporally manages/stores by data-compressing extracted subject information, the camcorder operation condition information, the background information, and the shot-to-shot relation information, video data, and all or a part of video taken spaces of a plurality of videos which are resynthesized by using the camcorder operation condition information, the background information, and the shot-to-shot relation information, the video data.

38. A method of spatio-temporally integrating/managing a plurality of videos according to claim 37, wherein the video structure information managing/storing step data-compresses the extracted subject information as the still images by developing the extracted subject information into the space, and data-compresses, as the still images, video taken spaces of a plurality of videos which are resynthesized.

39. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos, comprising:

an image data string saving step of reading video data to save the video data as data string in an image data string memory portion;

a camcorder operation condition detecting step of reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information;

a video splitting step of splitting videos of the data string into respective shots based on the camcorder on/off information;

a subject/background separating step of separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties;

a subject motion information extracting step of correlating subject information separated every frame between frames;

a video taken space resynthesizing step of resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame;

a shot-to-shot relation calculating step of calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing step respectively, based on a plurality of shots being separated; and

a video structure information managing/storing step of managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.

40. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, further comprising:

a video structure information transmitting/receiving step of transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.

41. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, further comprising, after the video structure information managing/storing step:

a resynthesizing step of resynthesizing one or plural video taken spaces and one or plural subjects based on stored/managed information in compliance with one or both of predetermined conditions and user'"'"'s requests; and

a display or outputting of displaying or outputting videos resynthesized by the resynthesizing step.

42. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, wherein the camcorder operation condition detecting step comprises,

a linear component calculating step of calculating respective horizontal/vertical linear components of image frames read by the image data string memory step,

an image data rearrangement step of rearranging spatio-temporal placements which are occupied by the images including calculated respective horizontal/vertical linear components,

a video information filter processing of applying filter process to the image data derived by rearrangement,

a feature extracting step of extracting features from results of the filter process, and

a statistical feature analyzing step of detecting the camcorder on/off information and the camcorder operation information by analyzing statistically the extracted features.

43. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 42, wherein the image data rearrangement step rearranges image data string into plural sheets of spatio-temporal sectional images which include normals of the image and a time axis.

44. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 42, wherein the video information filter step detects edges or lines of the video information of image data obtained by rearrangement.

45. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 44, wherein the feature extracting step extracts features by adding information as for detected edges or lines along a normal direction of the images.

46. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 42, wherein the statistical feature analyzing step detects straight lines of extracted features, which are represented as two-dimensional images having the time axis and the space axis, perpendicular to the time axis, to calculate the camcorder on/off information.

47. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 42, wherein the statistical feature analyzing step comprises,

a correlating step of correlating space coordinates by comparing spatial distributions at any two moments of extracted features which are represented on two-dimensional images having the time axes and the space axis, and

a camcorder operation parameter calculating step of calculating camcorder operation parameters by processing statistically correlated space coordinates.

48. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, wherein the subject/background separating step comprises,

a camcorder operation canceling step of canceling the camcorder operation between adjacent image data based on the camcorder operation information detected by the camcorder operation condition detecting step,

an image data comparing step of comparing the image data from which the camcorder operation information are canceled,

a comparison image data comparing step of comparing adjacent comparison image data, and

an area extracting step of extracting an subject area from comparison data which are calculated by the comparison image data comparing step.

49. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 48, wherein the camcorder operation canceling step deforms/displaces adjacent image frames based on the camcorder operation information to cancel change/displacement caused by the camcorder operation.

50. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 48, wherein the image data comparing step executes difference process of brightness and color information between adjacent image data from which the camcorder operations are canceled.

51. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 48, wherein the area extracting step comprises,

a binarization processing step of carrying out binarization process of the comparison data,

a labeling step of carrying out labeling binarization data which have been subjected to the binarization process,

a physical feature calculating step of calculating physical features in a labeled area, and

a collating step of collating calculated physical features with predetermined conditions to extract the subject area.

52. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 51, wherein the subject/background separating step extracts the background by subtracting the subject area per frame extracted by the area extracting step from the frame image.

53. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 51, wherein the subject motion information extracting step collates physical features of adjacent areas in time of the subject area extracted frame by frame by the area extracting step with predetermined conditions to correlate the subject information between the frames.

54. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, wherein the video taken space resynthesizing step superposes adjacent image frames into a continuous video taken space by deforming/displacing the adjacent image frames based on the camcorder operation information detected by the camcorder operation condition detecting portion.

55. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, wherein the shot-to-shot relation calculating step transforms a video taken space per shot generated by the video taken space resynthesizing portion so as to make size and positions thereof equal between video taken spaces.

56. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 39, wherein the video structure information managing/storing step spatio-temporally manages/stores by data-compressing extracted subject information, the camcorder operation condition information, the background information, and the shot-to-shot relation information, video data, and all or a part of video taken spaces of a plurality of videos which are resynthesized by using the camcorder operation condition information, the background information, and the shot-to-shot relation information, the video data.

57. A computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos according to claim 56, wherein the video structure information managing/storing step data-compresses the extracted subject information as the still images by developing the extracted subject information into the space, and data-compresses, as the still images, video taken spaces of a plurality of videos which are resynthesized.