Method and apparatus for refining target position and size estimates using image and depth data

• US 20050013465A1
• Filed: 07/10/2003
• Published: 01/20/2005
• Est. Priority Date: 07/10/2003
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hirvonen, David, Camus, Theodore Armand, Southall, John Benjamin
• Granted Patent: US 7,321,669 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A vehicle vision system that uses a depth map, image intensity data, and system calibration parameter to determine a target'"'"'s dimensions and relative position. Initial target boundary information is projected onto the depth map and onto the image intensity. A visibility analysis determines whether the rear of a target is within the system'"'"'s field of view. If so, the mapped image boundary is analyzed to determine an upper boundary of the target. Then, vertical image edges of the mapped image boundary are found by searching for a strongest pair of vertical image edges that are located at about the same depth. Then, the bottom of the mapped image boundary is found (or assumed from calibration parameters). Then, the target'"'"'s position is found by an averaging technique. The height and width of the target are then computed.

Claims

1. A method of refining a vision system comprising:

obtaining image intensity data of a field of view and a depth map of objects within the field of view;

determining if the image intensity data and the depth map include information related to a target;

searching for the top of the target in the image intensity data using information in the depth map;

searching for vertical edges of the target in the image intensity data using information in the depth map; and

searching for the bottom of the target in the image intensity data using information in the depth map.

2. A method according to claim 1, further including producing information related to the height and width of the target.

3. A method according to claim 2, further including refining the image of the vision system.

4. A method according to claim 2 wherein the produced height information is derived from system calibration parameters.

5. A method according to claim 2 wherein searching for the top of the target includes:

(a) searching a row of the image intensity data to determine whether a predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(b) storing the position of the row of step (a) if that row has the predetermined percentage of depth measurements that fall within a specified depth;

(c) searching a higher row of the image intensity data to determine whether the predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(d) looping to step (b) if the row of step (c) has the predetermined percentage of depth measurements that fall within the specified depth; and

(e) identifying the highest row that has the predetermined percentage of depth measurements that fall within the specified depth as the top of the target.

6. A method according to claim 2 wherein searching for the vertical edges includes:

searching for a pair of vertical image edges within the image intensity data;

determining if the found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth.

7. A method according to claim 6 wherein searching for the vertical edges further includes determining if the vertical image edges that are located at about the same depth and that are the strongest pair of vertical image edges meet maximum and minimum strength constraints.

8. A method according to claim 2 wherein searching for the bottom of the target includes searching lower rows of the image intensity data to find a dark-to-light transition.

9. A method according to claim 8 wherein the bottom of the target is set at a predetermined position if a dark-to-light transition is not found.

10. A method according to claim 5 further including:

searching for the vertical edges by;

searching for a pair of vertical image edges within the image intensity data;

determining if the found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth; and

searching for the bottom of the target by searching lower rows of the image intensity data to find a dark-to-light transition.

11. A method of refining a vision system comprising:

(a) obtaining image intensity data from a field of view, a depth map of objects within the field of view, and initial boundary information related to a target;

(b) mapping the initial boundary information onto the image intensity data and onto the depth map;

(c) searching near the mapped initial boundary conditions of the image intensity data and of the depth map to find the top of the target;

(d) searching near the mapped initial boundary conditions of the image intensity data and of the depth map to find vertical edges of the target; and

(e) searching near the mapped initial boundary conditions of the image intensity data and of the depth map to find the bottom of the target.

12. A method according to claim 11, further including using the mapped initial boundary conditions in the image intensity data and in the mapped depth map to refine the target'"'"'s position.

13. A method according to claim 11, further including producing information related to the height and width of the target.

14. A method according to claim 11 wherein searching for the top of the target includes:

(a) searching a row of the mapped image intensity data to determine whether a predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(b) storing the position of the row of step (a) if that row has the predetermined percentage of depth measurements that fall within a specified depth;

(c) searching the next higher row of the mapped image intensity data to determine whether the predetermined percentage of that row'"'"'s depth measurements fall within a specified depth;

(d) looping to step (b) if the row of step (c) has the predetermined percentage of depth measurements that fall within the specified depth; and

(e) identifying the highest row of the mapped image intensity data that has the predetermined percentage of depth measurements that fall within the specified depth as the top of the target.

15. A method according to claim 11 wherein searching for the vertical edges includes:

searching for a pair of vertical image edges within the mapped image intensity data;

determining if a found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth.

16. A method according to claim 15 wherein searching for the vertical edges further includes determining if the vertical image edges that are located at about the same depth and that are the strongest pair of vertical image edges meet maximum and minimum strength constraints.

17. A method according to claim 11 wherein searching for the bottom of the target includes searching lower rows of the mapped image intensity data to find a dark-to-light transition.

18. A method according to claim 17 wherein the bottom of the target is set at a predetermined position if a dark-to-light transition is not found.

19. A method according to claim 14 further including:

searching for the vertical edges by;

searching for a pair of vertical image edges within the mapped image intensity data;

determining if a found pair of vertical image edges are located at about the same depth; and

determining if the vertical image edges that are located at about the same depth are the strongest pair of vertical image edges, with the strength of a pair being the sum of the pixel intensity differences multiplied by a penalty term that reflects the difference in depth; and

searching for the bottom of the target by searching lower rows of the mapped image intensity data to find a dark-to-light transition.

20. A method of refining a vision system according to claim 11 wherein finding the top of the target includes:

analyzing a bottom row of the mapped initial boundary information in the image intensity data to determine whether a predetermined percentage of the bottom row'"'"'s depth measurements fall within a predetermined specified region around the target'"'"'s depth;

analyzing each consecutive higher row until the topmost row having the predetermined percentage of depth measurements that fall within the predetermined specified region around the target'"'"'s depth is found; and

relating that topmost row to the top of the target.

21. A method of refining a vision system according to claim 11 wherein finding the vertical edges of the target includes searching near the mapped initial boundary information of the image intensity data to find vertical image edges that are located at about the same depth.

22. An apparatus for refining a vision system comprising:

means for obtaining image intensity data from a field of view, a depth map of objects within the field of view, initial boundary information related to a target, and system calibration parameters;

means for mapping the initial boundary information onto the image intensity data and onto the depth map; and

means for determining from the mapped image intensity data and from the mapped depth map whether the specified portion of the target is within the field of view.

23. The apparatus according to claim 22, further including means for using the mapped initial boundary conditions of the image intensity data and of the depth map to refine information related to the target'"'"'s position.

24. The apparatus according to claim 22 wherein the means for searching to find the top of the target includes:

means for analyzing the bottom row of the mapped initial boundary information of the image intensity data to determine whether a predetermined percentage of the bottom row'"'"'s depth measurements fall within a predetermined specified region around the target'"'"'s depth;

means for analyzing each consecutive higher row until the topmost row having the predetermined percentage of depth measurements that fall within the predetermined specified region around the target'"'"'s depth is found; and

means for relating the topmost row to the top of the target.