Images background subtraction for dynamic lighting scenarios

• US 10,373,316 B2
• Filed: 04/20/2017
• Issued: 08/06/2019
• Est. Priority Date: 04/20/2017
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Jales Costa, Bruno Sielly, Goh, Madeline J
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US15/492,961
• Publication Number:
  US 20180308236A1
• Time in Patent Office: 838 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G08B 21/18 Status alarms G08B21/02 takes precedence
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A method and an apparatus for background subtraction highly applicable in autonomous driving scenarios are described. The method involves a reduction of illumination effects by constructing a normality background image from a normality model based on a plurality of baseline images taken under different illuminating conditions. A subtracted image is obtained by subtracting the normality background image from a scene image pixel-wise (i.e., pixel-by-pixel). The scene image may contain one or more foreground objects. The foreground objects are identified by highlighting the pixels in the subtracted image whose intensity is more than a predetermined standard deviation in the normality model. An illumination-invariant color space transformation algorithm may optionally be utilized to further reduce the variant illumination effects.

Claims

1. A method of sending a signal to a passenger of a transportation vehicle, comprising:

receiving a scene image of an interior of the transportation vehicle, the scene image containing a background and one or more foreground objects of interest;

projecting a normality background image of the interior from a normality model, wherein the normality background image comprises a plurality of background pixels that represents the background without the one or more foreground objects;

generating a subtracted image by subtracting the normality background image pixel-wise from the scene image;

highlighting a plurality of pixels in the subtracted image based on a standard deviation threshold to identify at least one of the one or more foreground objects as a foreign object to the transportation vehicle;

displaying the subtracted image including the plurality of highlighted pixels on a display device; and

generating a text message, a beeping sound or a flashing light as the signal to remind the passenger of the foreign object,

wherein the normality model comprises corresponding mean values and standard deviations of pixel intensity data associated with each of the plurality of background pixels of the normality background image,

wherein each of the plurality of pixels highlighted in the subtracted image has a corresponding pixel intensity that is more than the standard deviation threshold of the respective pixel, and

wherein the standard deviation threshold of the respective pixel is a multiple of the corresponding standard deviation associated with the respective pixel in the normality model.

2. The method of claim 1, further comprising:

constructing the normality model based on a plurality of baseline images taken under a plurality of illuminating conditions respectively, wherein each of the baseline images contains the background excluding the one or more foreground objects.

3. The method of claim 2, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein the constructing of the normality model comprises:

calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and

calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels.

4. The method of claim 2, wherein the mean values and the standard deviations comprise a Gaussian distribution or a student-t distribution that substantially fits the plurality of baseline images statistically.

5. The method of claim 1, wherein the receiving of the scene image comprises assessing an illuminating condition of the scene image, wherein the projecting of the normality background image is based on the illuminating condition, and wherein the standard deviation threshold is determined based on the illuminating condition.

6. A method, comprising:

receiving a scene image containing a background and one or more foreground objects of interest;

projecting a normality background image from a normality model, wherein the normality background image contains the background without the one or more foreground objects;

converting the scene image to a first illumination-invariant image using an illumination-invariant color space transformation algorithm;

converting the normality background image to a second illumination-invariant image using the illumination-invariant color space transformation algorithm;

generating a first subtracted image by subtracting the normality background image pixel-wise from the scene image;

generating a second subtracted image by subtracting the second illumination-invariant image pixel-wise from the first illumination-invariant image;

highlighting a plurality of pixels in the first subtracted image based on a standard deviation threshold to obtain a residual image as an identification of at least one of the one or more foreground objects;

applying a multiplying matrix operation between the residual image and the second subtracted image to generate an output image;

displaying the output image on a display device; and

generating a text message, a beeping sound or a flashing light in response to the identification.

7. The method of claim 6, wherein the normality background image comprises a plurality of background pixels, and wherein the normality model comprises corresponding mean values and standard deviations of pixel intensity data associated with each of the background pixels of the normality background image.

8. The method of claim 7, further comprising:

constructing the normality model based on a plurality of baseline images taken under a plurality of illuminating conditions respectively, wherein each of the baseline images contains the background excluding the one or more foreground objects.

9. The method of claim 8, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein the constructing of the normality model comprises:

calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and

calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels.

10. The method of claim 6, wherein the highlighting of the pixels in the first subtracted image based on the standard deviation threshold comprises highlighting the pixels in the first subtracted image having corresponding pixel intensity data more than the standard deviation threshold.

11. The method of claim 6, wherein the receiving of the scene image comprises assessing an illuminating condition of the scene image, wherein the projecting of the normality background image is based on the illuminating condition, and wherein the standard deviation threshold is determined based on the illuminating condition.

12. The method of claim 6, wherein the multiplying matrix operation takes the residual image and the second subtracted image of a same dimension as input to produce the output image with the same dimension, and wherein the multiplying matrix operation multiplies pixel intensity data of a same pixel location in the residual image and the second subtracted image respectively.

13. An apparatus implementable to an autonomous vehicle, comprising:

a camera configured to monitor a scene of interest containing a background; and

a processor communicatively coupled to the camera, the processor configured to perform acts comprising;

controlling the camera to take a plurality of baseline images of the scene under a plurality of illuminating conditions, wherein the baseline images are absent from one or more foreground objects;

controlling the camera to record a plurality of scene images;

creating a normality background image based on the baseline images;

generating a first subtracted image by subtracting the normality background image pixel-wise from one scene image of the plurality of scene images to determine whether the one or more foreground objects exist in the one scene image;

generating, via an autonomous driving system of the autonomous vehicle, a signal as a reminder to a passenger of the autonomous vehicle in response to an existence of the one or more foreground objects in the one scene image;

marking the one or more foreground objects in the first subtracted image as a residual image in response to the existence of the one or more foreground objects in the one scene image;

converting the one scene image to a first illumination-invariant image using an illumination-invariant color space transformation algorithm;

converting the normality background image to a second illumination-invariant image using the illumination-invariant color space transformation algorithm;

generating a second subtracted image by subtracting the second illumination-invariant image pixel-wise from the first illumination-invariant image;

applying a multiplying matrix operation between the residual image and the second subtracted image to generate an output image; and

displaying the output image on a display device of the autonomous vehicle to the passenger.

14. The apparatus of claim 13, wherein the normality background image comprises a plurality of background pixels, and wherein, in creating of the normality background image, the processor is configured to construct a normality model from the plurality of the baseline images taken under the plurality of illuminating conditions, the normality model comprising corresponding mean values and standard deviations of pixel intensity data associated with each of the background pixels of the normality background image.

15. The apparatus of claim 14, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein, in constructing the normality model, the processor is configured to perform acts comprising:

calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and

calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels.

16. The apparatus of claim 14, wherein the mean values and the standard deviations comprise a Gaussian distribution or a student-t distribution that substantially fits the plurality of baseline images statistically.

17. The apparatus of claim 13, wherein the processor is further configured to perform a marking of a plurality of pixels in the first subtracted image based on a standard deviation threshold, and wherein the marking of the pixels in the first subtracted image comprises highlighting the pixels having corresponding pixel intensity data more than the standard deviation threshold.

18. The apparatus of claim 13, wherein the multiplying matrix operation takes the residual image and the second subtracted image of a same dimension as input to produce the output image with the same dimension, and wherein the multiplying matrix operation multiplies pixel intensity data of a same pixel location in the residual image and the second subtracted image respectively.

19. The apparatus of claim 13, wherein the signal comprises a text message, a beeping sound, a flashing light, or a combination thereof.