Computer vision method and apparatus for imaging sensors for recognizing and tracking occupants in fixed environments under variable illumination

• US 6,608,910 B1
• Filed: 09/02/1999
• Issued: 08/19/2003
• Est. Priority Date: 09/02/1999
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Owechko, Yuri, Srinivasa, Narayan
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/389,874
• Time in Patent Office: 1,447 Days
• Field of Search: 382/100, 382/104, 382/154, 382/103, 382/278, 382/106, 340/436, 340/903, 340/540, 340/438, 359/843, 359/839, 359/872, 359/568, 356/375, 356/376, 356/139.03, 356/152.2
• US Class Current: 382/100
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

An computer vision method and system for recognizing and tracking occupants in a fixed space under variable illumination. The system utilizes a camera to capture an initial image of the unoccupied fixed space and subsequently captures images of the occupied fixed space. The edge maps of the current estimate of the unoccupied fixed space including illumination variations and the occupied fixed space are computed. The edge map of the current estimate of the unoccupied fixed space is then subtracted from the edge map of the occupied fixed space to yield a residual edge map, which is then processed to extract the image of the occupant. At least one equivalent rectangle is then computed from the two-dimensional moments of the image of the occupant. The equivalent rectangles are then used to determine the occupant type and position and to track changes in real-time. This method and system is generally designed for use with automobile safety systems such as “smart” airbags. However, it may be tailored to many applications such as computer gaming, adaptive automotive controls, and “smart” homes, among others.

Claims

1. A computer vision apparatus for recognizing and tracking occupants in a fixed space including:

at least one sensor fixedly positioned to capture light from a fixed space beginning at a time prior to an occupant'"'"'s entrance and continuing subsequent to an occupant'"'"'s entrance to form a series of N images having pixels with pixel intensities, the series of N images including a current image N_c and a previous image N<sub>c−

1</sub>, with the previous image N<sub>c−

1</sub> being the image immediately prior to the current image N_c in the series of N subsequent images, the sensor converting the series of N images into electronic data;

a processor positioned to receive electronic data from the sensor and to compare the current image N_c with the previous image N<sub>c−

1</sub> in order to dynamically determine the occupant type and occupant position;

computing an low-level vision-based feature map of the previous image N<sub>c −

1</sub>of the fixed space;

computing an low-level vision-based feature map of the current image N_c of the fixed space;

computing a residual low-level vision-based feature map having connected regions withsizes, by calculating the difference between low-level vision-based feature map of the current image N_c of the fixed space and the previous image N<sub>c−

1</sub> of the fixed space, where the pixels corresponding to the occupant in the previous subsequent image have been replaced by corresponding pixels of the current image N_c of the fixed space;

applying a high pass size filter to remove connected regions of the residual low-level vision-based feature map below a specified size, resulting in a size-filtered residual low-level-vision-based feature map including gaps and a bounded area;

filling the bounded area of the size-filtered residual low-level vision-based feature map, resulting in a filled low-level vision-based feature area;

calculating two-dimensional moments of the filled low-level vision-based feature area and using the two-dimensional moments to construct an equivalent rectangle with moments equal to that of the filled low-level vision-based feature area; and

determining the centroid, orientation, size and aspect ratio of the equivalent rectangle.

2. A computer vision apparatus for recognizing and tracking occupants in a fixed space as set forth in claim 1, further including at least one illumination source fixedly positioned to provide illumination to the fixed space and an occupant therein.

3. A computer vision apparatus for recognizing and tracking occupants in a fixed space as set forth in claim 2, wherein the illumination source is an IR source.

7. A computer vision apparatus for recognizing and tracking occupants in a fixed space as set forth in claim 1 wherein the processing system utilizes the method of claim 6.

9. A computer vision method for recognizing and tracking occupants in a fixed space as set forth in claim 7, further including the steps of:

(a) partitioning the filled edge area into a plurality N of sub-filled edge areas;

(b) calculating two-dimensional moments of each of the N sub-filled edge areas and using the two dimensional moments to construct an equivalent rectangle for each of the N sub-filled edge areas; and

(c) determining centroid, orientation, size, and aspect ratio of each of the equivalent rectangles for each of the N sub-filled edge areas.

11. A computer vision method for recognizing and tracking occupants in a fixed space as set forth in claim 3, further including the steps of:

(a) repeating steps (a) to (h) of claim 5 a plurality of times;

(b) computing the circumscribing circle of the equivalent rectangle during each repetition;

(c) comparing the filled edge area generated during the current repetition with the circumscribing circle of the equivalent rectangle computed during the previous repetition; and

(d) removing that part of the filled edge area falling outside the circumscribing circle the step of calculating the two-dimensional moments of the filled edge area and using the two-dimensional moments to construct an equivalent rectangle.

12. A computer vision apparatus for recognizing and tracking occupants in a fixed space as set forth in claim 1, wherein the low-level vision-based features are edges.

4. A computer vision method for recognizing and tracking occupants in a fixed space including the steps of:

(a) providing at least one sensor fixedly positioned to capture light reflected from a fixed space beginning at a time prior an occupant'"'"'s entrance and continuing subsequent to an occupant'"'"'s entrance to form a series of N images having pixels with pixel intensities, the series of N images including a current image N_c and a previous image N<sub>c−

1</sub>, with the previous image N<sub>c−

1</sub> being the image immediately prior to the current image N_c in the series of N subsequent images, the sensor converting the series of N images into electronic data; and

(b) processing the electronic data with a processing system positioned to receive electronic data from the sensor and to compare current image N_c with the previous image N<sub>c−

1</sub> in order to dynamically determine the occupant type and occupant position.

5. A computer vision method for recognizing and tracking occupants in a fixed space as set forth in claim 4 wherein the processing step (b) further includes the steps of:

(a) computing an edge map of the previous image N<sub>c−

1</sub> of the fixed space;

(b) computing an edge map of the current image N_c of the fixed space;

(c) computing a residual edge map having connected regions with sizes, by calculating the difference between edge map of the current image N_c of the fixed space and the previous image N<sub>c−

1</sub> of the fixed space, where the pixels corresponding to the occupant in the previous subsequent image have been replaced by corresponding pixels of the current image N_c of the fixed space;

(d) applying a high pass size filter to remove connected regions of the residual edge map below a specified size, resulting in a size-filtered residual edge map including gaps and a bounded area;

(e) filling the bounded area of the size-filtered residual edge map, resulting in a filled edge area;

(f) calculating two-dimensional moments of the filled edge area and using the two-dimensional moments to construct an equivalent rectangle with moments equal to that of the filled edge area;

(g) determining the centroid, orientation, size, and aspect ratio of the equivalent rectangle.

6. A computer vision method for recognizing and tracking occupants in a fixed space as set forth in claim 5, wherein the sensors include onboard edge extraction and two-dimensional moment computation capability.

8. A computer vision method for recognizing and tracking occupants in a fixed space as set forth in claim 6 wherein the filling step (e) further includes the steps of:

(a) applying an edge dilation operator to the size-filtered residual edge map to dilate the size-filtered residual edge map to eliminate gaps, resulting in a dilated residual edge map including edge pixels and a bounded area; and

(b) vertically filling the area between the edge pixels in the dilated residual edge map using the pixel intensity from the subsequent images resulting in a filled edge area.

10. A computer vision method for recognizing and tracking occupants in a fixed space including the steps of:

(a) providing at least one sensor fixedly positioned to capture light reflected from a fixed space beginning at a time prior to an occupant'"'"'s entrance and continuing subsequent to an occupant'"'"'s to form a series of N images having pixels with pixel intensities, the series of N images including a current image N_c and a previous image N<sub>c−

1</sub>, with the previous image N<sub>c−

1</sub> being the image immediately prior to the current image N_c in the series of N subsequent images, the sensor converting the series of N images into electronic data; and

(b) processing the electronic data with a processing system positioned to receive electronic data from the sensor and to;

i. compute an edge map of the previous image N<sub>c−

1</sub> of the fixed space;

ii. compute an edge map of the current image N_c of the fixed space;

iii. compute a residual edge map having connected regions with sizes, from the difference between the edge map of the current image N_c of the fixed space and the previous image N<sub>c−

1</sub> of the fixed space, where the pixels corresponding to the occupant in the previous subsequent image have been replaced by corresponding pixels of the current image N_c of the fixed space;

iv. apply a high pass size filter to remove connected regions of the residual edge map below a specified size, resulting in a size-filtered residual edge map including gaps;

v. apply an edge dilation operator to the size-filtered residual edge map to dilate the size-filtered residual edge map to eliminate gaps, resulting in a dilated residual edge map including edge pixels;

vi. vertically fill the area between the edge pixels in the dilated residual edge map using the pixel intensity from the subsequent images resulting in a filled edge area;

vii. calculate two-dimensional moments of the filled edge area and using the two-dimensional moments to construct an equivalent rectangle with moments equal to that of the filled edge area;

viii. determine the centroid, orientation, size, and aspect ratio of the equivalent rectangle;

ix. partitioning the filled edge area into a plurality N of sub-filled edge areas;

x. calculating the two-dimensional moments of each of the N sub-filled edge areas and using the two dimensional moments to construct an equivalent rectangle for each of the N sub-filled edge areas; and

xi. determining the centroid, orientation, size, and aspect ratio of each of the equivalent rectangles for each of the N sub-filled edge areas; and

(c) providing the centroid, orientation, size, and aspect ratio of the equivalent rectangle and the centroid, orientation, size, and aspect ratio of each of the equivalent rectangles for each of the N sub-filled edge areas as output.