Method and apparatus for refining target position and size estimates using image and depth data

• US 7,321,669 B2
• Filed: 07/10/2003
• Issued: 01/22/2008
• Est. Priority Date: 07/10/2003
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Ford Motor Company, Sarnoff Corporation (SRI International, Inc.)
• Inventors: Southall, John Benjamin, Hirvonen, David, Camus, Theodore Armand
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Akhavannik, Hadi
• Application Number:
  US10/617,231
• Publication Number:
  US 20050013465A1
• Time in Patent Office: 1,657 Days
• Field of Search: 382103-108, 340/904, 340/961, 701/301
• US Class Current: 382/106
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A vehicle vision system that uses a depth map, image intensity data, and system calibration parameter to determine a target'"'"'s dimensions and relative position. Initial target boundary information is projected onto the depth map and onto the image intensity. A visibility analysis determines whether the rear of a target is within the system'"'"'s field of view. If so, the mapped image boundary is analyzed to determine an upper boundary of the target. Then, vertical image edges of the mapped image boundary are found by searching for a strongest pair of vertical image edges that are located at about the same depth. Then, the bottom of the mapped image boundary is found (or assumed from calibration parameters). Then, the target'"'"'s position is found by an averaging technique. The height and width of the target are then computed.

Claims

1. A method of refining a vision system comprising:

obtaining image intensity data of a field of view and a depth map of objects within the field of view;

determining if the image intensity data and the depth map include information related to a target;

searching for vertical edges of the target in the image intensity data using information in the depth map;

searching for the bottom of the target in the image intensity data using information in the depth map; and

searching for the top of the target in the image intensity data using information in the depth map, which is aligned to the image intensity data, wherein said searching for the top of the target comprises;

(a) searching a row of the depth map data to determine whether a predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(b) storing the position of the row of step (a) if that row has the predetermined percentage of depth measurements that fall within a specified depth;

(c) searching a higher row of the depth map data to determine whether the predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(d) looping to step (b) if the row of step (c) has the predetermined percentage of depth measurements that fall within the specified depth; and

(e) identifying the highest row that has the predetermined percentage of depth measurements that fall within the specified depth as the top of the target.

2. A method according to claim 1, further comprising producing information related to the height and width of the target in the intensity image and depth map data.

3. A method according to claim 2, further comprising refining the depth information the target.

4. A method according to claim 2 wherein the produced height and width information in meters is derived from system calibration parameters.

5. A method according to claim 1 wherein searching for the bottom of the target comprise searching the lower rows of the image intensity data in the region of the detected target to find a dark-to-light transition.

6. A method according to claim 5 wherein the bottom of the target is set at a predetermined position if a dark-to-light transition is not found.

7. A method according to claim 1 further comprising:

searching for the vertical edges by;

searching for a pair of vertical image edges within the image intensity data;

determining if the found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth; and

searching for the bottom of the target by searching lower rows of the image intensity data to find a dark-to-light transition.

8. A method of refining a vision system comprising:

obtaining image intensity data from a field of view, a depth map of objects within the field of view, and initial boundary information related to a target;

mapping the initial boundary information onto the image intensity data and onto the depth map;

searching near the mapped initial target boundary of the image intensity data and of the depth map to find vertical edges of the target;

searching near the mapped initial target boundary of the image intensity data and of the depth map to find the bottom of the target; and

searching near the mapped initial target boundary of the image intensity data and of the depth map to find the top of the target, wherein said searching for the top of the target comprises;

(a) searching a row of the mapped depth map data to determine whether a predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(b) storing the position of the row of step (a) if that row has the predetermined percentage of depth measurements that fall within a specified depth;

(c) searching the next higher row of the mapped depth map data to determine whether the predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(d) looping to step (b) if the row of step (c) has the predetermined percentage of depth measurements that fall within the specified depth; and

(e) identifying the highest row of the mapped depth map data that has the predetermined percentage of depth measurements that fall within the specified depth as the top of the target.

9. A method according to claim 8, further comprising using the mapped initial target boundary in the image intensity data and in the mapped depth map to refine the target'"'"'s position.

10. A method according to claim 8, further comprising producing information related to the height and width of the target.

11. A method according to claim 8 wherein searching for the bottom of the target comprise searching lower rows of the mapped image intensity data to find a dark-to-light transition.

12. A method according to claim 11 wherein the bottom of the target is set at a predetermined position if a dark-to-light transition is not found.

13. A method according to claim 8 further comprising:

searching for the vertical edges by;

searching for a pair of vertical image edges within the mapped image intensity data;

determining if a found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth; and

searching for the bottom of the target by searching lower rows of the mapped image intensity data to find a dark-to-light transition.

14. A method of refining a vision system according to claim 8 wherein finding the top of the target comprises:

analyzing a bottom row of the mapped initial boundary information in the depth map data to determine whether a predetermined percentage of the bottom row'"'"'s depth measurements fall within a predetermined specified region around the target'"'"'s depth;

analyzing each consecutive higher row until the topmost row having the predetermined percentage of depth measurements that fall within the predetermined specified region around the target'"'"'s depth is found; and

relating that topmost row to the top of the target.

15. A method of refining a vision system according to claim 8 wherein finding the vertical edges of the target comprises searching near the mapped initial boundary information of the image intensity data to find vertical image edges that are located at about the same depth.

16. An apparatus for refining a vision system comprising:

means for obtaining image intensity data from a field of view, a depth map of objects within the field of view, initial boundary information related to a target, and system calibration parameters;

means for mapping the initial boundary information onto the image intensity data and onto the depth map; and

means for determining from the mapped image intensity data and from the mapped depth map whether the specified portion of the initial target boundary is wholly within the field of view, said means for determining comprising means for searching to find top of the target comprising means for analyzing the bottom row of the mapped initial boundary information of the depth map data to determine whether a predetermined percentage of the bottom row'"'"'s depth measurements fall within a predetermined specified region around the target'"'"'s depth, means for analyzing each consecutive higher row until the topmost row having the predetermined percentage of depth measurements that fall within the predetermined specified region around the target'"'"'s depth is found and means for relating the topmost row to the top of the target.

17. The apparatus according to claim 16, further comprising means for using the mapped initial target boundary of the image intensity data and of the depth map to refine information related to the target'"'"'s position.

18. The apparatus of claim 16 wherein said means for determining further comprising means for searching to find bottom of the target.

19. The apparatus of claim 16 wherein said means for determining further comprising means for searching to find vertical edges of the target.

20. A method of refining a vision system comprising:

obtaining image intensity data of a field of view and a depth map of objects within the field of view;

determining if the image intensity data and the depth map include information related to a target;

searching for the top of the target in the image intensity data using information in the depth map, which is aligned to the image intensity data;

searching for the bottom of the target in the image intensity data using information in the depth map; and

searching for vertical edges of the target in the image intensity data using information in the depth map;

wherein said searching for the vertical edges of the target comprises searching for a pair of vertical image edges within the image intensity data, determining if the found pair of vertical image edges are located at about the same depth, and determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth.

21. The method according to claim 20 wherein searching for the vertical edges further comprises determining if the vertical image edges that are located at about the same depth and that are the strongest pair of vertical image edges meet maximum and minimum strength constraints.

22. A method of refining a vision system comprising:

obtaining image intensity data from a field of view, a depth map of objects within the field of view, and initial boundary information related to a target;

mapping the initial boundary information onto the image intensity data and onto the depth map;

searching near the mapped initial target boundary of the image intensity data and of the depth map to find the top of the target;

searching near the mapped initial target boundary of the image intensity data and of the depth map to find the bottom of the target; and

searching near the mapped initial target boundary of the image intensity data and of the depth map to find vertical edges of the target, wherein said searching for the vertical edges of the target comprise searching for a pair of vertical image edges within the mapped image intensity data, determining if a found pair of vertical image edges are located at about the same depth, and determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth.

23. The method according to claim 22 wherein searching for the vertical edges further comprises determining if the vertical image edges that are located at about the same depth and that are the strongest pair of vertical image edges meet maximum and minimum strength constraints.