Image processing device

• US 7,139,019 B2
• Filed: 05/24/2002
• Issued: 11/21/2006
• Est. Priority Date: 06/05/2001
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sawao, Takashi, Ishibashi, Junichi, Kondo, Tetsujiro, Nagano, Takahiro, Wada, Seiji, Miyake, Toru, Fujiwara, Naoki
• Primary Examiner(s): Natnael, Paulos M.
• Application Number:
  US10/343,675
• Publication Number:
  US 20040021775A1
• Time in Patent Office: 1,642 Days
• Field of Search: 348/222.1, 348/699, 348/443, 348/460, 348/461, 348/701, 348/345, 348/169, 382292-293, 382/103, 382/48, 382/107
• US Class Current: 348/222.1
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

The present invention enables a movement vector of a moving object to the detected, by detecting a mixture ratio indicating the proportion of mixing with an image, or taking into consideration a region mixed with the image. A weighted frame difference computation unit 3423 outputs weighted difference. A movement compensation unit 3424 and a correlation value computing unit 3427 compute the correlation between a block of interest containing each pixel of interest of weighted difference image data, and a corresponding block of weighted difference image data of an adjacent frame. A maximum value judgment unit 3428 detects at least one of weight and a movement vector wherein correlation between weighted difference image data is greatest. The present invention can be applied to signals processing device s for processing images.

Claims

1. An image processing device for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said image processing device comprising:

weighted difference image data calculating means for calculating a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on weighting which is to correspond to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and outputting as weighted difference image data corresponding to said frame of interest;

inter-weighted-difference-image correlation data computing means for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector which is to indicate relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest containing each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for, with regard to each predetermined increment of said inter-weighted-difference-image correlation data, detecting at least one of said weighting and said movement vector where the correlation between said weighted difference image data is maximum, setting said weighting to said mixture ratio corresponding to said increment of said frame of interest in the event that said weighting is detected, and setting said movement vector to said movement vector corresponding to said increment of said frame of interest in the event that said movement vector is detected.

2. An image processing device according to claim 1, further comprising:

weighting information generating means for generating a plurality of said weights corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

wherein said weighted difference image data calculating means detect said weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computes the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, and outputs as said mixture ratio corresponding to said increment of said frame of interest.

3. An image processing device according to claim 1, further comprising:

region specifying means for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

mixture ratio estimating means for estimating said ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information; and

estimated movement vector generating means for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

wherein said weighted difference image data calculating means calculate difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data, and output as first difference image data, and calculate difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data, and output as second difference image data;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computes the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputs as inter-difference-image correlation data;

and wherein said detecting means detect said estimated movement vector where the correlation is maximum, based on said inter-weighted-difference-image correlation data, and outputs as said movement vector corresponding to said pixel of interest of said frame of interest.

4. An image processing device according to claim 1, further comprising:

weighting information generating means for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights; and

movement vector information generating means for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein said weighted difference image data calculating means calculate said weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and outputs as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, compute the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputs as inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, sets said detected weight to said mixture ratio corresponding to said increment of said frame of interest, sets said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputs at least one of said mixture ratio and said movement vector.

5. An image processing device according to claim 1, further comprising:

weighting information generating means for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating said mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating the weight of each of the generated weights; and

movement vector information generating means for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein said weighted difference image data calculating means calculate weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame of said image data, and outputs as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computes the correlation between said block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputs as inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, sets said detected weight to said mixture ratio corresponding to said increment of said frame of interest, sets said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputs at least one of said mixture ratio and said movement vector.

6. An image processing method for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said method comprising:

a weighted difference image data calculating step for calculating a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on weighting which is to correspond to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data computing step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector which is to indicate relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest containing each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for, with regard to each predetermined increment of said inter-weighted-difference-image correlation data, detecting at least one of said weighting and said movement vector where the correlation between said weighted difference image data is maximum, setting said weighting to said mixture ratio corresponding to said increment of said frame of interest in the event that said weighting is detected, and setting said movement vector to said movement vector corresponding to said increment of said frame of interest in the event that said movement vector is detected.

7. An image processing method according to claim 6, further comprising:

a weighting information generating step for generating a plurality of said weights corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

wherein, in said weighted difference image data calculating step, said weighted difference is detected based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, the correlation is calculated between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data;

and wherein in said detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, and output as said mixture ratio corresponding to said increment of said frame of interest.

8. An image processing method according to claim 6, further comprising:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

a mixture ratio estimating step for estimating said mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information; and

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

wherein, in said weighted difference image data calculating step, difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data, and output as first difference image data, and difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data, and output as second difference image data;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said first difference image data and said second difference image data is matched based on said estimated movement vector, the correlation is computed between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and output as inter-difference-image correlation data;

and wherein, in said detecting step, said estimated movement vector where the correlation is maximum is detected based on said inter-weighted-difference-image correlation data, and output as said movement vector corresponding to said pixel of interest of said frame of interest.

9. An image processing method according to claim 6, further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, said weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

10. An image processing method according to claim 6, further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating said mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

11. A recording medium storing a computer-readable program for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said program comprising:

a weighted difference image data calculating step for calculating a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on weighting which is to correspond to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data computing step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector which is to indicate relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest containing each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for, with regard to each predetermined increment of said inter-weighted-difference-image correlation data, detecting at least one of said weighting and said movement vector where the correlation between said weighted difference image data is maximum, setting said weighting to said mixture ratio corresponding to said increment of said frame of interest in the event that said weighting is detected, and setting said movement vector to said movement vector corresponding to said increment of said frame of interest in the event that said movement vector is detected.

12. A recording medium according to claim 11, said program further comprising:

a weighting information generating step for generating a plurality of said weights corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

wherein, in said weighted difference image data calculating step, said weighted difference is detected based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, the correlation is computed between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data;

and wherein in said detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, and output as said mixture ratio corresponding to said increment of said frame of interest.

13. A recording medium according to claim 11, said program further comprising:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

a mixture ratio estimating step for estimating said mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information; and

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

wherein, in said weighted difference image data calculating step, difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data, and output as first difference image data, and difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data, and output as second difference image data;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said first difference image data and said second difference image data is matched based on said estimated movement vector, the correlation is computed between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and output as inter-difference-image correlation data;

and wherein, in said detecting step, said estimated movement vector where the correlation is maximum is detected based on said difference image correlation data, and output as said movement vector corresponding to said pixel of interest of said frame of interest.

14. A recording medium according to claim 11, said program further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weight; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, said weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

15. A recording medium according to claim 11, said program further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

16. A program for causing a computer, which processes image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, to execute:

a weighted difference image data calculating step for calculating a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on weighting which is to correspond to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data computing step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector which is to indicate relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest containing each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for, with regard to each predetermined increment of said inter-weighted-difference-image correlation data, detecting at least one of said weighting and said movement vector where the correlation between said weighted difference image data is maximum, setting said weighting to said mixture ratio corresponding to said increment of said frame of interest in the event that said weighting is detected, and setting said movement vector to said movement vector corresponding to said increment of said frame of interest in the event that said movement vector is detected.

17. A program according to claim 16, further comprising:

a weighting information generating step for generating a plurality of said weights corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

wherein, in said weighted difference image data calculating step, said weighted difference is detected based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, the correlation is computed between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data;

and wherein in said detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, and output as said mixture ratio corresponding to said increment of said frame of interest.

18. A program according to claim 16, further comprising:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

a mixture ratio estimating step for estimating said mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information; and

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

wherein, in said weighted difference image data calculating step, difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data, and output as first difference image data, and difference is calculated based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data, and output as second difference image data;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said first difference image data and said second difference image data is matched based on said estimated movement vector, the correlation is computed between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and output as inter-difference-image correlation data;

and wherein, in said detecting step, said estimated movement vector where the correlation is maximum is detected based on said inter-weighted-difference-image correlation data, and output as said movement vector corresponding to said pixel of interest of said frame of interest.

19. A program according to claim 16, further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, said weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

20. A program according to claim 16, further comprising:

a weighting information generating step for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights; and

a movement vector information generating step for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein, in said weighted difference image data calculating step, weighted difference is calculated based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data computing step, the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame is matched according to said movement vector indicated by said movement vector information, the correlation is computed between said block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data;

and wherein, in said detecting step, said weighting and said movement vector where the correlation between said weighted difference image data is maximum are detected for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, said detected weight is set to said mixture ratio corresponding to said increment of said frame of interest, said detected movement vector is set to said movement vector corresponding to said increment of said frame of interest, and at least one of said mixture ratio and said movement vector is output.

21. An image-taking device, comprising:

image-taking means for outputting a subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects as image data made up of a predetermined number of pieces of pixel data;

weighted difference image data calculating means for calculating a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on weighting which is to correspond to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and outputting as weighted difference image data corresponding to said frame of interest;

inter-weighted-difference-image correlation data computing means for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector which is to indicate relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest containing each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for, with regard to each predetermined increment of said inter-weighted-difference-image correlation data, detecting at least one of said weighting and said movement vector where the correlation between said weighted difference image data is maximum, setting said weighting to said mixture ratio corresponding to said increment of said frame of interest in the event that said weighting is detected, and setting said movement vector to said movement vector corresponding to said increment of said frame of interest in the event that said movement vector is detected.

22. An image-taking device according to claim 21, further comprising:

weighting information generating means for generating a plurality of said weights corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

wherein said weighted difference image data calculating means detect said weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputs as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computes the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputs as said inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, and outputs as said mixture ratio corresponding to said increment of said frame of interest.

23. An image-taking device according to claim 21, further comprising:

region specifying means for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

mixture ratio estimating means for estimating said mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information; and

estimated movement vector generating means for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

wherein said weighted difference image data calculating means calculate difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data, and output as first difference image data, and calculate difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data, and output as second difference image data;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computes the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputs as inter-difference-image correlation data;

and wherein said detecting means detect said estimated movement vector where the correlation is maximum, based on said inter-weighted-difference-image correlation data, and outputs as said movement vector corresponding to said pixel of interest of said frame of interest.

24. An image-taking device according to claim 21, further comprising:

weighting information generating means for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating a mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights; and

movement vector information generating means for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein said weighted difference image data calculating means calculate said weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and outputs as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computes the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputs as inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, sets said detected weight to said mixture ratio corresponding to said increment of said frame of interest, sets said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputs at least one of said mixture ratio and said movement vector.

25. An image-taking device according to claim 21, further comprising:

weighting information generating means for generating a plurality of said weights each with different values, corresponding to said mixture ratio indicating said mixed state in said pixel data of said objects which are multiple in the real world, and generating weighting information indicating each of said generated weights; and

movement vector information generating means for generating a plurality of said movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

wherein said weighted difference image data calculating means calculate weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame adjacent to said frame of interest of said image data, and outputs as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computes the correlation between said block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputs as inter-weighted-difference-image correlation data;

and wherein said detecting means detect said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, sets said detected weight to said mixture ratio corresponding to said increment of said frame of interest, sets said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputs at least one of said mixture ratio and said movement vector.

26. An image processing device for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said image processing device comprising:

weighting information generating means for generating a plurality of weights corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

weighted difference image data detecting means for detecting a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on said weight indicated by said weighting information, and outputting as weighted difference image data corresponding to said frame of interest;

inter-weighted-difference-image correlation data detecting means for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

mixture ratio detecting means for, with regard to each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, detecting said weighting where the correlation between said weighted difference image data is maximum, and outputting as said mixture ratio corresponding to said increment of said frame of interest.

27. An image processing device according to claim 26, wherein said weighting information generating means sequentially generate said weighting information indicating a plurality of said weights which each differ, corresponding to a plurality of said mixture ratios which each differ;

and wherein said weighted difference image data detecting means detect said weighted difference corresponding to said plurality of weights which each differ, and output as weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data detecting means compute said correlation corresponding to said plurality of weights which each differ, and output as inter-weighted-difference-image correlation data;

and wherein said mixture ratio detecting means detect, from said plurality of weights which each differ, said weighting where the correlation between said weighted difference image data is maximum, and output as said mixture ratio corresponding to said increment of said frame of interest.

28. An image processing device according to claim 26, wherein said inter-weighted-difference-image correlation data detecting means compute, as said correlation, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image. correlation data.

29. An image processing device according to claim 26, wherein said inter-weighted-difference-image correlation data detecting means compute, as said correlation, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

30. An image processing device according to claim 26, wherein said weighted difference image data detecting means detect a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and detect a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

31. An image processing device according to claim 26, further comprising region information generating means for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

32. An image processing device according to claim 26, further comprising movement vector detecting means for detecting said movement vector indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame.

33. An image processing device according to claim 26, wherein said mixture ratio detecting means detect said mixture ratio of said pixel of said frame of interest, belonging to a mixed region where said plurality of objects are mixed.

34. An image processing device according to claim 26, further comprising separating means for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

35. An image processing device according to claim 34, further comprising blurring adjustment means for adjusting the amount of movement blurring of objects making up said separated foreground.

36. An image processing device according to claim 34, further comprising synthesizing means for synthesizing desired other objects and objects making up said separated foreground, based on said mixture ratio.

37. An image processing method for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said method comprising:

a weighting information generating step for generating a plurality of weights corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data detecting step for detecting a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on said weight indicated by said weighting information, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data detecting step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a mixture ratio detecting step for, with regard to each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, detecting said weighting where the correlation between said weighted difference image data is maximum, and outputting as said mixture ratio corresponding to said increment of said frame of interest.

38. An image processing method according to claim 37, wherein, in said weighting information generating step, said weighting information indicating a plurality of said weights which each differ is sequentially generated, corresponding to a plurality of said mixture ratios which each differ;

and wherein, in said weighted difference image data detecting step, said weighted difference is detected corresponding to said plurality of weights which each differ, and output as weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data detecting step, said correlation corresponding to said plurality of weights which each differ is computed, and output as inter-weighted-difference-image correlation data;

and wherein, in said mixture ratio detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected from said plurality of weights which each differ, and output as said mixture ratio corresponding to said predetermined increment of said frame of interest.

39. An image processing method according to claim 37, wherein, in said inter-weighted-difference-image correlation data detecting step, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as inter-weighted-difference-image correlation data.

40. An image processing method according to claim 37, wherein, in said inter-weighted-difference-image correlation data detecting step, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as said inter-weighted-difference-image correlation data.

41. An image processing method according to claim 37, wherein, in said weighted difference image data detecting step, a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest is detected in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest is detected in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

42. An image processing method according to claim 37, further comprising a region information generating step for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

43. An image processing method according to claim 37, further comprising a movement vector detecting step for detecting said movement vector indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame.

44. An image processing method according to claim 37, wherein, in said mixture ratio detecting step, said mixture ratio of said pixel of said frame of interest belonging to a mixed region where said plurality of objects are mixed is detected.

45. An image processing method according to claim 37, further comprising a separating step for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

46. An image processing method according to claim 45, further comprising a movement blurring adjustment step for adjusting the amount of movement blurring of objects making up said separated foreground.

47. An image processing method according to claim 45, further comprising a synthesizing step for synthesizing desired other objects and objects making up said separated foreground, based on said mixture ratio.

48. A recording medium storing a computer-readable program for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said program comprising:

a weighting information generating step for generating a plurality of weights corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data detecting step for detecting a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on said weight indicated by said weighting information, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data detecting step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a mixture ratio detecting step for, with regard to each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, detecting said weighting where the correlation between said weighted difference image data is maximum, and outputting as said mixture ratio corresponding to said increment of said frame of interest.

49. A recording medium according to claim 48, wherein, in said weighting information generating step, said weighting information indicating a plurality of said weights which each differ is sequentially generated, corresponding to a plurality of said mixture ratios which each differ;

and wherein, in said weighted difference image data detecting step, said weighted difference is detected corresponding to said plurality of weights which each differ, and output as said weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data detecting step, said correlation corresponding to said plurality of weights which each differ is computed, and output as inter-weighted-difference-image correlation data;

and wherein, in said mixture ratio detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected from said plurality of weights which each differ, and output as said mixture ratio corresponding to said predetermined increment of said frame of interest.

50. A recording medium according to claim 48, wherein, in said inter-weighted-difference-image correlation data detecting step, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as inter-weighted-difference-image correlation data.

51. A recording medium according to claim 48, wherein, in said inter-weighted-difference-image correlation data detecting step, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as said inter-weighted-difference-image correlation data.

52. A recording medium according to claim 48, wherein, in said weighted difference image data detecting step, a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest is detected in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest is detected in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

53. A recording medium according to claim 48, said program further comprising a region information generating step for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

54. A recording medium according to claim 48, said program further comprising a movement vector detecting step for detecting said movement vector indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame.

55. A recording medium according to claim 48, wherein, in said mixture ratio detecting step, said mixture ratio of said pixel of said frame of interest belonging to a mixed region where said plurality of objects are mixed is detected.

56. A recording medium according to claim 48, said program further comprising a separating step for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

57. A recording medium according to claim 56, said program further comprising a movement blurring adjustment step for adjusting the amount of movement blurring of objects making up said separated foreground.

58. A recording medium according to claim 56, said program further comprising a synthesizing step for synthesizing desired other objects and objects making up said separated foreground, based on said mixture ratio.

59. A program for causing a computer, which processes image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, to execute:

a weighting information generating step for generating a plurality of weights corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data detecting step for detecting a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on said weight indicated by said weighting information, and outputting as weighted difference image data corresponding to said frame of interest;

an inter-weighted-difference-image correlation data detecting step for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a mixture ratio detecting step for, with regard to each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, detecting said weighting where the correlation between said weighted difference image data is maximum, and outputting as said mixture ratio corresponding to said increment of said frame of interest.

60. A program according to claim 59, wherein, in said weighting information generating step, said weighting information indicating a plurality of said weights which each differ is sequentially generated, corresponding to a plurality of said mixture ratios which each differ;

and wherein, in said weighted difference image data detecting step, said weighted difference is detected corresponding to said plurality of weights which each differ, and output as weighted difference image data corresponding to said frame of interest;

and wherein, in said inter-weighted-difference-image correlation data detecting step, said correlation corresponding to said plurality of weights which each differ is computed, and output as inter-weighted-difference-image correlation data;

and wherein, in said mixture ratio detecting step, said weighting where the correlation between said weighted difference image data is maximum is detected from said plurality of weights which each differ, and output as said mixture ratio corresponding to said increment of said frame of interest.

61. A program according to claim 59, wherein, in said inter-weighted-difference-image correlation data detecting step, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as inter-weighted-difference-image correlation data.

62. A program according to claim 59, wherein, in said inter-weighted-difference-image correlation data detecting step, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame is computed as said correlation, and output as inter-weighted-difference-image correlation data.

63. A program according to claim 59, wherein, in said weighted difference image data detecting step, a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest is detected in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest is detected in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

64. A program according to claim 59, further comprising a region information generating step for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

65. A program according to claim 59, further comprising a movement vector detecting step for detecting said movement vector indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame.

66. A program according to claim 59, wherein, in said mixture ratio detecting step, said mixture ratio of said pixel of said frame of interest belonging to a mixed region where said plurality of objects are mixed is detected.

67. A program according to claim 59, further comprising a separating step for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

68. A program according to claim 67, further comprising a movement blurring adjustment step for adjusting the amount of movement blurring of objects making up said separated foreground.

69. A program according to claim 67, further comprising a synthesizing step for synthesizing desired other objects and objects making up said separated foreground, based on said mixture ratio.

70. An image-taking device, comprising:

image-taking means for outputting a subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects as image data made up of a predetermined number of pieces of pixel data;

weighting information generating means for generating a plurality of weights corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

weighted difference image data detecting means for detecting a weighted difference between each pixel in a frame of interest in said image data and each pixel in an adjacent frame which is adjacent to said frame of interest in said image data, based on said weight indicated by said weighting information, and outputting as weighted difference image data corresponding to said frame of interest;

inter-weighted-difference-image correlation data detecting means for matching a relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to a movement vector for indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, computing the correlation between a block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and a corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

mixture ratio detecting means for, with regard to each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, detecting said weighting where the correlation between said weighted difference image data is maximum, and outputting as said mixture ratio corresponding to said increment of said frame of interest.

71. An image-taking device according to claim 70, wherein said weighting information generating means sequentially generate said weighting information indicating a plurality of said weights which each differ, corresponding to a plurality of said mixture ratios which each differ;

and wherein said weighted difference image data detecting means detect said weighted difference corresponding to said plurality of weights which each differ, and output as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data detecting means compute said correlation corresponding to said plurality of weights which each differ, and output as inter-weighted-difference-image correlation data;

and wherein said mixture ratio detecting means detect, from said plurality of weights which each differ, said weighting where the correlation between said weighted difference image data is maximum, and output as said mixture ratio corresponding to said predetermined increment of said frame of interest.

72. An image-taking device according to claim 70, wherein said inter-weighted-difference-image correlation data detecting means compute, as said correlation, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

73. An image-taking device according to claim 70, wherein said inter-weighted-difference-image correlation data detecting means compute, as said correlation, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

74. An image-taking device according to claim 70, wherein said weighted difference image data detecting means detect a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and detect a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

75. An image-taking device according to claim 70, further comprising region information generating means for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

76. An image-taking device according to claim 70, further comprising movement vector detecting means for detecting said movement vector indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame.

77. An image-taking device according to claim 70, wherein said mixture ratio detecting means detect said mixture ratio of said pixel of said frame of interest, belonging to a mixed region where said plurality of objects are mixed.

78. An image-taking device according to claim 70, further comprising separating means for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

79. An image-taking device according to claim 78, further comprising movement blurring adjustment means for adjusting the amount of movement blurring of objects making up said separated foreground.

80. An image-taking device according to claim 78, further comprising synthesizing means for synthesizing desired other objects and objects making up said separated foreground, based on said mixture ratio.

81. An image processing device for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said image processing device comprising:

region specifying means for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating the specified mixed region;

mixture ratio estimating means for estimating mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information;

estimated movement vector generating means for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

difference image data calculating means for calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data and outputting as first difference image data, and calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data and outputting as second difference image data;

inter-difference-image correlation data computing means for matching the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computing the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputting as inter-difference-image correlation data; and

detecting means for detecting said estimated movement vector where the correlation is maximum, based on said inter-difference-image correlation data, and outputting as said movement vector corresponding to said pixel of interest of said frame of interest.

82. An image processing device according to claim 81, wherein said inter-difference-image correlation data computing means compute, as said correlation, a difference absolute value between said block of interest and said corresponding block, and output as inter-difference-image correlation data.

83. An image processing device according to claim 81, wherein said inter-difference-image correlation data computing means compute, as said correlation, a sum of absolute value of difference between said block of interest and said corresponding block, and output as inter-weighted-difference-image correlation data.

84. An image processing device according to claim 81, further comprising separating means for separating, from said pixel data, at least said foreground objects, based on said mixture ratio.

85. An image processing device according to claim 84, further comprising movement blurring adjustment means for adjusting the amount of movement blurring of said separated foreground objects, based on said movement vector.

86. An image processing device according to claim 84, further comprising noise removing means for removing noise of said separated foreground objects, based on said movement vector.

87. An image processing method for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said method comprising:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

a mixture ratio estimating step for estimating mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information;

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

a difference image data calculating step for calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data and outputting as first difference image data, and calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data and outputting as second difference image data;

an inter-difference-image correlation data computing step for matching the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computing the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputting as inter-difference-image correlation data; and

a detecting step for detecting said estimated movement vector where the correlation is maximum, based on said inter-difference-image correlation data, and outputting as said movement vector corresponding to said pixel of interest of said frame of interest.

88. A recording medium storing a computer-readable image processing program for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said program comprising:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating the specified mixed region;

a mixture ratio estimating step for estimating mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information;

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

a difference image data calculating step for calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of a frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data and outputting as first difference image data, and calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data and outputting as second difference image data;

an inter-difference-image correlation data computing step for matching the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computing the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputting as inter-difference-image correlation data; and

a detecting step for detecting said estimated movement vector where the correlation is maximum, based on said inter-difference-image correlation data, and outputting as said movement vector corresponding to said pixel of interest of said frame of interest.

89. A program for causing a computer, which processes image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, to execute:

a region specifying step for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

a mixture ratio estimating step for estimating mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information;

an estimated movement vector generating step for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

a difference image data calculating step for calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data and outputting as first difference image data, and calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data and outputting as second difference image data;

an inter-difference-image correlation data computing step for matching the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computing the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputting as inter-difference-image correlation data; and

a detecting step for detecting said estimated movement vector where the correlation is maximum, based on said inter-difference-image correlation data, and outputting as said movement vector corresponding to said pixel of interest of said frame of interest.

90. An image-taking device, comprising:

image-taking means for outputting a subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects as image data made up of a predetermined number of pieces of pixel data;

region specifying means for specifying a mixed region made up of a mixture of a foreground object component configuring foreground objects and a background object component configuring background objects based on said image data, and outputting region information indicating said specified mixed region;

mixture ratio estimating means for estimating mixture ratio indicating ratio of mixture between said foreground object component and said background object component in said pixel data belonging to said mixed region, based on said image data and said region information;

estimated movement vector generating means for generating a plurality of estimated movement vectors, and outputting each of said estimated movement vectors that have been generated;

difference image data calculating means for calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a first adjacent frame adjacent to said frame of interest of said image data and outputting as first difference image data, and calculating difference based on said mixture ratio of said pixel of interest of said frame of interest between each pixel of said frame of interest of said image data and each pixel of a second adjacent frame adjacent to said frame of interest of said image data and outputting as second difference image data;

inter-difference-image correlation data computing means for matching the relative position of said first difference image data and said second difference image data based on said estimated movement vector, computing the correlation between said block of interest made up of at least one pixel centered a pixel of said first difference image data corresponding to said pixel of interest of said frame of interest, and said corresponding block made up of at least one pixel of said second difference image data, and outputting as inter-difference-image correlation data; and

detecting means for detecting said estimated movement vector where the correlation is maximum, based on said inter-difference-image correlation data, and outputting as said movement vector corresponding to said pixel of interest of said frame of interest.

91. An image-taking device according to claim 90, wherein said inter-difference-image correlation data computing means compute, as said correlation, a difference absolute value between said block of interest and said corresponding block, and output as said inter-difference-image correlation data.

92. An image-taking device according to claim 90, wherein said inter-difference-image correlation data computing means compute, as said correlation, a sum of absolute value of difference between said block of interest and said corresponding block, and output as inter-weighted-difference-image correlation data.

93. An image-taking device according to claim 90, further comprising separating means for separating, from said pixel data, at least said foreground objects, based on said mixture ratio.

94. An image-taking device according to claim 93, further comprising movement blurring adjustment means for adjusting the amount of movement blurring of said separated foreground objects, based on said movement vector.

95. An image-taking device according to claim 93, further comprising noise removing means for removing noise of said separated foreground objects, based on said movement vector.

96. An image processing device for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said image processing device comprising:

weighting information generating means for generating a plurality of weights each with differing values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

weighted difference image data calculating means for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

movement vector information generating means for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

inter-weighted-difference-image correlation data computing means for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

97. An image processing device according to claim 96, wherein said inter-weighted-difference-image correlation data computing means match the relative position of at least said weighted difference image data of said frame of interest, said weighted difference image data of a frame before said frame of interest, and said weighted difference image data of a frame following said frame of interest, according to said movement vector indicated by said movement vector information, compute the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest, a corresponding block made up of at least one pixel of said weighted difference image data of said previous frame, and a corresponding block made up of at least one pixel of said weighted difference image data of said following frame, and output as inter-weighted-difference-image correlation data.

98. An image processing device according to claim 96, wherein said inter-weighted-difference-image correlation data computing means compute, as said correlation, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data.

99. An image processing device according to claim 96, wherein said inter-weighted-difference-image correlation data computing means compute, as said correlation, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data.

100. An image processing device according to claim 96, wherein said weighted difference image data calculating means calculate a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and calculate a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

101. An image processing device according to claim 96, comprising region information generating means for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

102. An image processing device according to claim 96, wherein said detecting means detect said mixture ratio of said pixel of said frame of interest, belonging to a mixed region where said plurality of objects are mixed.

103. An image processing device according to claim 96, further comprising separating means for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

104. An image processing device according to claim 103, further comprising movement blurring adjustment means for adjusting the amount of movement blurring of objects making up said separated foreground.

105. An image processing device according to claim 103, further comprising noise removing means for removing noise of said separated foreground objects, based on said movement vector.

106. An image processing method for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said method comprising:

a weighting information generating step for generating a plurality of weights each with differing values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

107. A recording medium storing a computer-readable image processing program for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said program comprising:

a weighting information generating step for generating a plurality of weights each with differing values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

108. A program for causing a computer, which processes image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, to execute:

a weighting information generating step for generating a plurality of weights each with differing values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

109. An image-taking device, comprising:

image-taking means for outputting a subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects as image data made up of a predetermined number of pieces of pixel data;

weighting information generating means for generating a plurality of weights each with differing values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating said weight of each of the generated weights;

weighted difference image data calculating means for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of said frame of interest of said image data and each pixel of said adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

movement vector information generating means for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

inter-weighted-difference-image correlation data computing means for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of at least one pixel of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

110. An image-taking device according to claim 109, wherein said inter-weighted-difference-image correlation data computing means match the relative position of at least said weighted difference image data of said frame of interest, said weighted difference image data of a frame before said frame of interest, and said weighted difference image data of a frame following said frame of interest, according to said movement vector indicated by said movement vector information, compute the correlation between said block of interest made up of at least one pixel centered on each pixel of interest of said weighted difference image data of said frame of interest, a corresponding block made up of at least one pixel of said weighted difference image data of said previous frame, and a corresponding block made up of at least one pixel of said weighted difference image data of said following frame, and output as inter-weighted-difference-image correlation data.

111. An image-taking device according to claim 109, wherein said inter-weighted-difference-image correlation data computing means compute, as said correlation, a difference absolute value between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data.

112. An image-taking device according to claim 109, wherein said inter-weighted-difference-image correlation data computing means compute, as said correlation, a sum of absolute value of difference between said block of interest made up of at least one pixel and centered on each pixel of interest of said weighted difference image data of said frame of interest and said corresponding block made up of at least one pixel of said weighted difference image data of said adjacent frame, and output as said inter-weighted-difference-image correlation data.

113. An image-taking device according to claim 109, wherein said weighted difference image data calculating means calculate a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame prior to said frame of interest in the event that said pixel of said frame of interest belongs to a covered background region formed at a tip side in the direction of movement of an object in the foreground and which is a mixed region wherein a plurality of said objects are mixed, and calculate a weighted difference due to said weight indicated by said weighting information between each pixel of said frame of interest and each pixel of said frame following said frame of interest in the event that said pixel of said frame of interest belongs to an uncovered background region formed at a trailing side in the direction of movement of an object in said foreground and which is a mixed region wherein a plurality of said objects are mixed.

114. An image-taking device according to claim 109, comprising region information generating means for specifying a foreground region made up of objects forming a foreground of said plurality of objects in said image data, a background region made up of objects forming a background of said plurality of objects in said image data, a covered background region formed at a tip side in the direction of movement of said foreground objects which is a mixed region wherein a plurality of said objects are mixed, and an uncovered background region formed at a trailing side in the direction of movement of said foreground objects which is said mixed region, and generating region information indicating said foreground region, said background region, and said mixed region including said covered background region and said uncovered background region.

115. An image-taking device according to claim 109, wherein said detecting means detect said mixture ratio of said pixel of said frame of interest, belonging to a mixed region where said plurality of objects are mixed.

116. An image-taking device according to claim 109, further comprising separating means for separating, from said pixel data, at least objects making up a foreground from said plurality of objects, based on said mixture ratio.

117. An image-taking device according to claim 116, further comprising movement blurring adjustment means for adjusting the amount of movement blurring of objects making up said separated foreground.

118. An image-taking device according to claim 116, further comprising noise removing means for removing noise of said separated foreground objects, based on said movement vector.

119. An image processing device for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said image processing device comprising:

weighting information generating means for generating a plurality of weights each with different values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

weighted difference image data calculating means for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of a frame of interest of said image data and each pixel of an adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

movement vector information generating means for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

inter-weighted-difference-image correlation data computing means for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between a first block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and a first corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

120. An image processing device according to claim 119, wherein said weighted difference image data calculating means calculate weighted difference, based on said weighting indicated by said weighting information between a second block of interest made up of a plurality of pixels centered on each pixel of interest of said frame of interest of said image data and a second corresponding block made up of a plurality of pixels of said adjacent frame adjacent to said frame of interest of said image data, with said weighting corresponding to said plurality of pixels of said second block of interest or said second corresponding block as being constant, and output as weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, compute the correlation between said first block of interest of said weighted difference image data of said frame of interest and said first corresponding block of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

121. An image processing device according to claim 119, wherein said weighted difference image data calculating means calculate weighted difference, based on said weighting corresponding to each pixel of said plurality of pixels indicated by said weighting information which is sequentially generated, between a second block of interest made up of a plurality of pixels centered on each pixel of interest of a frame of interest of said image data and a second corresponding block made up of a plurality of pixels of an adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, compute the correlation between said first block of interest of said weighted difference image data of said frame of interest and said first corresponding block of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

122. An image processing device according to claim 121, wherein said weighted difference image data calculating means calculate weighted difference based on said weighting which continuously changes corresponding to the position of each pixel, as well as corresponding to each pixel of said plurality of pixels indicated by said weighting information which is sequentially generated between said second block of interest and said second corresponding block, and output as weighted difference image data corresponding to said pixel of interest.

123. An image processing device according to claim 121, wherein said detecting means detect most accurate said weight and said movement vector of said plurality of said weights and said movement vectors by evaluating the continuity of said weighting corresponding to each pixel within said first block of interest where the correlation between said weighted difference image data is maximum, set said detected weight to said mixture ratio corresponding to said increment of said frame of interest, set said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and output at least one of said mixture ratio and said movement vector.

124. An image processing method for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said method comprising:

a weighting information generating step for generating a plurality of weights each with different values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of a frame of interest of said image data and each pixel of an adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between a first block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and a first corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

125. A recording medium storing a computer-readable image processing program for processing image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, said program comprising:

a weighting information generating step for generating a plurality of weights each with different values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of a frame of interest of said image data and each pixel of an adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between a first block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and a first corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

126. A program for causing a computer, which processes image data made up of a predetermined number of pieces of pixel data obtained by an image-taking device having a predetermined number of pixels having time-integration effects, to execute:

a weighting information generating step for generating a plurality of weights each with different values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

a weighted difference image data calculating step for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of a frame of interest of said image data and each pixel of an adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

a movement vector information generating step for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

an inter-weighted-difference-image correlation data computing step for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between a first block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and a first corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

a detecting step for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

127. An image-taking device, comprising:

image-taking means for outputting a subject image taken by an image-taking device having a predetermined number of pixels having time-integration effects as image data made up of a predetermined number of pieces of pixel data;

weighting information generating means for generating a plurality of weights each with different values, corresponding to a mixture ratio indicating a mixed state in said pixel data of objects which are multiple in the real world, and generating weighting information indicating each of said generated weights;

weighted difference image data calculating means for calculating weighted difference based on said weighting indicated by said weighting information between each pixel of a frame of interest of said image data and each pixel of an adjacent frame which is adjacent to said frame of interest of said image data, and outputting as weighted difference image data corresponding to said frame of interest;

movement vector information generating means for generating a plurality of movement vectors each having different values indicating relative movement between said pixel data of said frame of interest and said pixel data of said adjacent frame, and generating movement vector information indicating each of said generated movement vectors;

inter-weighted-difference-image correlation data computing means for matching the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, computing the correlation between a first block of interest made up of a plurality of pixels centered on each pixel of interest of said weighted difference image data of said frame of interest and a first corresponding block made up of a plurality of pixels of said weighted difference image data of said adjacent frame, and outputting as inter-weighted-difference-image correlation data; and

detecting means for detecting said weighting and said movement vector where the correlation between said weighted difference image data is maximum, for each predetermined increment made up of a plurality of pixels of said inter-weighted-difference-image correlation data, setting said detected weight to said mixture ratio corresponding to said increment of said frame of interest, setting said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and outputting at least one of said mixture ratio and said movement vector.

128. An image-taking device according to claim 127, wherein said weighted difference image data calculating means calculate weighted difference, based on said weighting indicated by said weighting information between a second block of interest made up of a plurality of pixels centered on each pixel of interest of said frame of interest of said image data and a second corresponding block made up of a plurality of pixels of said adjacent frame adjacent to said frame of interest of said image data, with said weighting corresponding to said plurality of pixels of said second block of interest or said second corresponding block as being constant, and output as weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, compute the correlation between said first block of interest of said weighted difference image data of said frame of interest and said first corresponding block of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

129. An image-taking device according to claim 127, wherein said weighted difference image data calculating means calculate weighted difference, based on said weighting corresponding to each pixel of said plurality of pixels indicated by said weighting information which is sequentially generated, between a second block of interest made up of a plurality of pixels centered on each pixel of interest of a frame of interest of said image data and a second corresponding block made up of a plurality of pixels of an adjacent frame adjacent to said frame of interest of said image data, and output as said weighted difference image data corresponding to said frame of interest;

and wherein said inter-weighted-difference-image correlation data computing means match the relative position of said weighted difference image data of said frame of interest and said weighted difference image data of said adjacent frame according to said movement vector indicated by said movement vector information, compute the correlation between said first block of interest of said weighted difference image data of said frame of interest and said first corresponding block of said weighted difference image data of said adjacent frame, and output as inter-weighted-difference-image correlation data.

130. An image-taking device according to claim 129, wherein said weighted difference image data calculating means calculate weighted difference based on said weighting which continuously changes corresponding to the position of each pixel, as well as corresponding to each pixel of said plurality of pixels indicated by said weighting information which is sequentially generated, between said second block of interest and said second corresponding block, and output as weighted difference image data corresponding to said pixel of interest.

131. An image-taking device according to claim 129, wherein said detecting means detect most accurate said weight and said movement vector of said plurality of said weights and said movement vectors by evaluating the continuity of said weighting corresponding to each pixel within said first block of interest where the correlation between said weighted difference image data is maximum, set said detected weight to said mixture ratio corresponding to said increment of said frame of interest, set said detected movement vector to said movement vector corresponding to said increment of said frame of interest, and output at least one of said mixture ratio and said movement vector.