Method of judging background/foreground position relationship between moving subjects and method of converting two-dimensional images into three-dimensional images

• US 5,699,443 A
• Filed: 02/12/1996
• Issued: 12/16/1997
• Est. Priority Date: 09/22/1994
• Status: Expired due to Term

• Current Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Iinuma, Toshiya, Uchida, Hidekazu, Murata, Haruhiko, Okino, Toshiyuki, Yamashita, Shugo, Tanase, Susumu
• Primary Examiner(s): JOHNS, ANDREW W
• Application Number:
  US08/600,052
• Time in Patent Office: 673 Days
• Field of Search: 382/100, 382/103, 382/107, 382/154, 382/173, 382/276, 348/25, 348/44, 348/155, 348/169, 348/699
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects

Abstract

The present invention relates to a method of judging the back-and-forth positional relationship between subjects. Amounts of horizontal movement of the subjects which are detected for detecting areas set in an image area constituting a 2D image are compared after being assigned plus or minus signs depending on the direction of the movement, thereby to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas. A target area is set in a peripheral portion in the image area, to judge which of the total number of first detecting areas existing in the target area and the total number of second detecting areas existing in the target area is larger. It is judged that the subjects in all the detecting areas whose total number existing in the target area is judged to be larger are positioned behind the subjects in all the detecting areas whose total number existing in the target area is judged to be smaller.

Claims

1. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas;

a fourth step of setting a target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said target area and the total number of second detecting areas existing in the target area is larger; and

a fifth step of judging that the subjects in all the detecting areas whose total number existing in the target area is judged to be larger are positioned behind the subjects in all the detecting areas whose total number existing in the target area is judged to be smaller.

2. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas;

a fourth step of setting a target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said target area and the total number of second detecting areas existing in the target area is larger; and

a fifth step of judging that the subjects in all the detecting areas whose total number existing in the target area is judged to be larger are positioned ahead of the subjects in all the detecting areas whose total number existing in the target area is judged to be smaller.

3. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas;

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger;

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger; and

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step.

4. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas;

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger;

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger;

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step;

a seventh step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas are the same, which of the total number of first detecting areas existing in all the detecting areas and the total number of second detecting areas existing in all the detecting areas is smaller; and

an eighth step of taking as third detecting areas the detecting areas whose total number existing in all the detecting areas is judged to be smaller in said seventh step out of the first detecting areas and the second detecting areas, to judge the back-and-forth positional relationship between the subjects in the first detecting areas and the subjects in the second detecting areas on the basis of the ratio of the total number of third detecting areas existing in said first target area to the total number of third detecting areas existing in all the detecting areas.

5. The method according to claim 4, wherein said eighth step comprises the steps of

judging that the subjects in one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned behind the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas existing in all the detecting areas is not less than a predetermined value, and

judging that the subjects in the one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned ahead of the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas in all the detecting areas is less than a predetermined value.

6. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas;

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger;

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger;

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step;

a seventh step of judging, when it is judged in said fourth step that the total number of either first detecting areas or second detecting areas existing in the first target area is larger and it is judged in said fifth step that the total numbers of first detecting areas and second detecting areas existing in the second target area are equal, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subject in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step;

an eighth step of judging, when it is judged in said fourth step that the total numbers of first detecting areas and second detecting areas existing in the first target area are equal and it is judged in said fifth step that the total number of either first detecting areas or second detecting areas existing in the second target area is larger, that the subjects in all the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step are positioned ahead of the subjects in all the detecting areas whose total number existing in the second target area is judged to be smaller in said fifth step;

a ninth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas are the same, or it is judged in both said fourth step and said fifth step that the total numbers of first detecting areas and second detecting areas in said target area are equal, which of the total number of first detecting areas existing in all the detecting areas and the total number of second detecting areas existing in all the detecting areas is smaller; and

a tenth step of taking as third detecting areas the detecting areas whose total number existing in all the detecting areas is judged to be smaller in said ninth step out of the first detecting areas and the second detecting areas, to judge the back-and-forth positional relationship between the subjects in the first detecting areas and the subjects in the second detecting areas on the basis of the ratio of the total number of third detecting areas existing in said first target area to the total number of third detecting areas existing in all the detecting areas.

7. The method according to claim 6, wherein said tenth step comprises the steps of

judging that the subjects in one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned behind the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas existing in all the detecting areas is not less than a predetermined value, and

judging that the subjects in the one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned ahead of the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas in all the detecting areas is less than a predetermined value.

8. A method of judging the back-and-forth positional relationship between subjects, comprising:

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects;

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas;

a third step of comparing for two detecting areas adjacent to each other in the horizontal direction out of said detecting areas the amounts of horizontal movement of the subjects which are detected in both the detecting areas;

a fourth step of judging, when the amounts of horizontal movement of the subjects which are detected in the two detecting areas adjacent to each other in the horizontal direction differ and the subject whose amount of horizontal movement is larger is moving toward the subject whose amount of horizontal movement is smaller, whether the amount of horizontal movement of a subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger;

a fifth step of judging, when it is judged in said fourth step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger, that the subject whose amount of horizontal movement is larger is positioned ahead of the subject whose amount of horizontal movement is smaller;

a sixth step of judging, when it is judged in said fourth step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is not changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger, that the subject whose amount of horizontal movement is larger is positioned behind the subject whose amount of horizontal movement is smaller;

a seventh step of judging, when the amounts of horizontal movement of the subjects which are detected in the two detecting areas adjacent to each other in the horizontal direction differ and the subject whose amount of horizontal movement is larger is not moving toward the subject whose amount of horizontal movement is smaller, whether the amount of horizontal movement of a subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is larger is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller;

an eighth step of judging, when it is judged in said seventh step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is larger is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller, that the subject whose amount of horizontal movement is larger is positioned ahead of the subject whose amount of horizontal movement is smaller; and

a ninth step of judging, when it is judged in said seventh step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is larger is detected is not changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller, that the subject whose amount of horizontal movement is larger is positioned behind the subject whose amount of horizontal movement is smaller.

9. A method of converting two-dimensional images into three-dimensional images, in which a main image signal and a sub-image signal delayed from the main image signal are produced from a two-dimensional image signal, and one of the main image signal and the sub-image signal is taken as a left eye image and the other image signal is taken as a right eye image, comprising the steps of:

judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller on the basis of said two-dimensional image signal; and

determining which of the main image signal and the sub-image signal is taken as a left eye image or a right eye image on the basis of the result of the judgment of the back-and-forth positional relationship.

10. The method according to claim 9, wherein

said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller comprises

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas,

a fourth step of setting a target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said target area and the total number of second detecting areas existing in the target area is larger, and

a fifth step of judging that the subjects in all the detecting areas whose total number existing in the target area is judged to be larger are positioned behind the subjects in all the detecting areas whose total number existing in the target area is judged to be smaller.

11. The method according to claim 9, wherein

said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller comprises

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas,

a fourth step of setting a target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said target area and the total number of second detecting areas existing in the target area is larger, and

a fifth step of judging that the subjects in all the detecting areas whose total number existing in the target area is judged to be larger are positioned ahead of the subjects in all the detecting areas whose total number existing in the target area is judged to be smaller.

12. The method according to claim 9, wherein

said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller comprises

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas,

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger,

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger, and

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step.

13. The method according to claim 9, wherein

said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller comprises

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas,

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger,

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger, and

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step,

a seventh step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas are the same, which of the total number of first detecting areas existing in all the detecting areas and the total number of second detecting areas existing in all the detecting areas is smaller, and

an eighth step of taking as third detecting areas the detecting areas whose total number existing in all the detecting areas is judged to be smaller in said seventh step out of the first detecting areas and the second detecting areas, to judge the back-and-forth positional relationship between the subjects in the first detecting areas and the subjects in the second detecting areas on the basis of the ratio of the total number of third detecting areas existing in said first target area to the total number of third detecting areas existing in all the detecting areas.

14. The method according to claim 13, wherein said eighth step comprises the steps of

judging that the subjects in one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned behind the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas existing in all the detecting areas is not less than a predetermined value, and

judging that the subjects in the one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned ahead of the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas in all the detecting areas is less than a predetermined value.

15. The method according to claim 9, wherein said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller, comprises

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing the amounts of horizontal movement of the subjects which are detected for said respective detecting areas after assigning the amounts of horizontal movement plus or minus signs depending on the direction of the movement, to classify all the detecting areas into two types of detecting areas in such a manner that the detecting areas where a relatively large amount of horizontal movement is detected are taken as first detecting areas, while the detecting areas where a relatively small amount of horizontal movement is detected are taken as second detecting areas,

a fourth step of setting a first target area in a peripheral portion in said image area, to judge which of the total number of first detecting areas existing in said first target area and the total number of second detecting areas existing in the first target area is larger,

a fifth step of setting a second target area in a central portion in said image area, to judge which of the total number of first detecting areas existing in said second target area and the total number of second detecting areas existing in the second target area is larger,

a sixth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas differ, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step,

a seventh step of judging, when it is judged in said fourth step that the total number of either first detecting areas or second detecting areas existing in the first target area is larger and it is judged in said fifth step that the total numbers of first detecting areas and second detecting areas existing in the second target area are equal, that the subjects in all the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step are positioned behind the subjects in all the detecting areas whose total number existing in the first target area is judged to be smaller in said fourth step,

an eighth step of judging, when it is judged in said fourth step that the total numbers of first detecting areas and second detecting areas existing in the first target area are equal and it is judged in said fifth step that the total number of either first detecting areas or second detecting areas existing in the second target area is larger, that the subjects in all the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step is positioned ahead of the subjects in all the detecting areas whose total number existing in the second target area is judged to be smaller in said fifth step,

a ninth step of judging, when the detecting areas whose total number existing in the first target area is judged to be larger in said fourth step out of the first detecting areas and the second detecting areas and the detecting areas whose total number existing in the second target area is judged to be larger in said fifth step out of the first detecting areas and the second detecting areas are the same, or it is judged in both said fourth step and said fifth step that the total numbers of first detecting areas and second detecting areas in said target area are equal, which of the total number of first detecting areas existing in all the detecting areas and the total number of second detecting areas existing in all the detecting areas is smaller, and

a tenth step of taking as third detecting areas the detecting areas whose total number existing in all the detecting areas is judged to be smaller in said ninth step out of the first detecting areas and the second detecting areas, to judge the back-and-forth positional relationship between subjects in the first detecting areas and the subjects in the second detecting areas on the basis of the ratio of the total number of third detecting areas existing in said first target area to the total number of third detecting areas existing in all the detecting areas.

16. The method according to claim 15, wherein said tenth step comprises the steps of

judging that the subjects in one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned behind the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas existing in all the detecting areas is not less than a predetermined value, and

judging that the subjects in the one detecting areas, which correspond to the third detecting areas, out of the first detecting areas and the second detecting areas are positioned ahead of the subjects in the other detecting areas when the ratio of the number of third detecting areas existing in the first target area to the total number of third detecting areas in all the detecting areas is less than a predetermined value.

17. The method according to claim 9 wherein

said step of judging the back-and-forth positional relationship between subjects whose amount of horizontal movement is larger and subjects whose amount of horizontal movement is smaller, comprising

a first step of setting in an image area constituting a two-dimensional image a plurality of detecting areas for detecting amounts of horizontal movement of the subjects,

a second step of respectively detecting the amounts of horizontal movement of the subjects for said detecting areas,

a third step of comparing for the two detecting areas adjacent to each other in the horizontal direction out of said detecting areas the amounts of horizontal movement of the subjects which are detected in both the detecting areas,

a fourth step of judging, when the amounts of horizontal movement of the subjects which are detected in the two detecting areas adjacent to each other in the horizontal direction differ and the subject whose amount of horizontal movement is larger is moving toward the subject whose amount of horizontal movement is smaller, whether the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger,

a fifth step of judging, when it is judged in said fourth step that the amount of horizontal movement of a subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger, that the subject whose amount of horizontal movement is larger is positioned ahead of the subject whose amount of horizontal movement is smaller,

a sixth step of judging, when it is judged in said fourth step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is smaller is detected is not changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is larger, that the subject whose amount of horizontal movement is larger is positioned behind the subject whose amount of horizontal movement is smaller,

a seventh step of judging, when the amounts of movement of the subjects which are detected in the two detecting areas adjacent to each other in the horizontal direction differ and the subject whose amount of horizontal movement is larger is not moving toward the subject whose amount of horizontal movement is smaller, whether the amount of horizontal movement of a subject which is detected after an elapse of a predetermined time period in the detecting areas where the subject whose amount of horizontal movement is larger is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller,

an eighth step of judging, when it is judged in said seventh step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is larger is detected is changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller, that the subject whose amount of horizontal movement is larger is positioned ahead of the subject whose amount of horizontal movement is smaller, and

a ninth step of judging, when it is judged in said seventh step that the amount of horizontal movement of the subject which is detected after an elapse of a predetermined time period in the detecting area where the subject whose amount of horizontal movement is larger is detected is not changed in the direction in which it approaches the amount of horizontal movement of the subject whose amount of horizontal movement is smaller, that the subject whose amount of horizontal movement is larger is positioned behind the subject whose amount of horizontal movement is smaller.