Image processing apparatus and image processing method for estimating three-dimensional position of object in image

• US 10,063,843 B2
• Filed: 09/09/2016
• Issued: 08/28/2018
• Est. Priority Date: 09/15/2015
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Yano, Kotaro, Uchiyama, Hiroyuki
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US15/261,601
• Publication Number:
  US 20170076454A1
• Time in Patent Office: 718 Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

An image processing apparatus includes an association unit, a storage unit, a first estimation unit, and a second estimation unit. The association unit associates detection results of a first object and a second object. The storage unit stores a positional relationship between respective fields of view of a first image capturing unit and a second image capturing unit. The first estimation unit estimates a three-dimensional position of the first object based on the positional relationship and the first object detected in respective images captured by the first image capturing unit and the second image capturing unit. The second estimation unit estimates a three-dimensional position of the second object based on the three-dimensional position of the first object associated by the association unit.

Claims

1. An image processing apparatus comprising:

one or more processors; and

a memory coupled to the one or more processors, the memory having stored thereon instructions which, when executed by the one or more processors, cause the apparatus to;

acquire an image from a first image capturing unit and a second image capturing unit;

detect a first object and a second object in the image;

associate detection results of the first object and the second object;

store a positional relationship between respective fields of view of the first image capturing unit and the second image capturing unit;

estimate a three-dimensional position of the first object based on the positional relationship and the first object detected in respective images captured by the first image capturing unit and the second image capturing unit; and

estimate a three-dimensional position of the second object based on the three-dimensional position of the associated first object.

2. The image processing apparatus according to claim 1, further comprising determine whether the second object has been detected in the respective images acquired from the first image capturing unit and the second image capturing unit, wherein

the estimate estimates the three-dimensional position of the second object in a case where the second object has not been detected in the image acquired from at least one of the first image capturing unit and the second image capturing unit.

3. The image processing apparatus according to claim 2, wherein the associate associates the detection results of the first object and the second object in a case where the second object has not been detected in the image acquired from at least one of the first image capturing unit and the second image capturing unit.

4. The image processing apparatus according to claim 1, further comprising recognize motion of the second object from the three-dimensional positions of the second object at a plurality of time points.

5. The image processing apparatus according to claim 1, wherein the first object and the second object are different regions of one object.

6. The image processing apparatus according to claim 5, wherein the first object and the second object are regions of a person.

7. The image processing apparatus according to claim 6, wherein the first object is a head of the person and the second object is a hand of the person.

8. The image processing apparatus according to claim 7, further comprising recognize motion of the hand of the person from the three-dimensional positions of the second object at a plurality of time points.

9. The image processing apparatus according to claim 8, wherein the motion of the hand of the person is motion of the person picking up an article with the hand.

10. The image processing apparatus according to claim 1, wherein the associate associates detection results of the first object and the second object based on respective detection positions.

11. The image processing apparatus according to claim 1, wherein the associate associates detection results of the first object and the second object based on respective degrees of likelihood.

12. The image processing apparatus according to claim 1, wherein the estimate estimates a distance to the second object from a distance to the first object.

13. An image processing method, comprising:

acquiring an image from a first image capturing unit and a second image capturing unit;

detecting a first object and a second object in the image;

associating detection results of the first object and the second object;

estimating a three-dimensional position of the first object based on the positional relationship between respective fields of view of the first image capturing unit and the second image capturing unit and the first object detected in respective images captured by the first image capturing unit and the second image capturing unit; and

estimating a three-dimensional position of the second object based on the three-dimensional position of the first object associated in the associating.

14. The image processing method according to claim 13, further comprising:

determining whether the second object has been detected in the respective images acquired from the first image capturing unit and the second image capturing unit; and

estimating the three-dimensional position of the second object in a case where the second object has not been detected in the image acquired from at least one of the first image capturing unit and the second image capturing unit.

15. The image processing method according to claim 13, further comprising recognizing motion of the second object from the three-dimensional positions of the second object at a plurality of time points.

16. The image processing method according to claim 13, wherein the first object and the second object are different regions of one object.

17. A non-transitory storage medium having stored therein a program causing a computer to execute each step of an image processing method, the method comprising:

acquiring an image from a first image capturing unit and a second image capturing unit;

detecting a first object and a second object in the image;

associating detection results of the first object and the second object;

estimating a three-dimensional position of the first object based on the positional relationship between respective fields of view of the first image capturing unit and the second image capturing unit and the first object detected in respective images captured by the first image capturing unit and the second image capturing unit; and

estimating a three-dimensional position of the second object based on the three-dimensional position of the first object associated in the associating.

18. The non-transitory storage medium according to claim 17, further comprising:

determining whether the second object has been detected in the respective images acquired from the first image capturing unit and the second image capturing unit; and

estimating the three-dimensional position of the second object in a case where the second object has not been detected in the image acquired from at least one of the first image capturing unit and the second image capturing unit.

19. The non-transitory storage medium according to claim 17, further comprising recognizing motion of the second object from the three-dimensional positions of the second object at a plurality of time points.

20. The non-transitory storage medium according to claim 17, wherein the first object and the second object are different regions of one object.