Method and system for video processing to remove noise from a digital video sequence containing a modulated light signal

• US 9,888,203 B2
• Filed: 04/07/2016
• Issued: 02/06/2018
• Est. Priority Date: 07/26/2011
• Status: Active Grant

• Current Assignee: ABL IP Holding LLC (Acuity Brands, Inc.)
• Original Assignee: ABL IP Holding LLC (Acuity Brands, Inc.)
• Inventors: Ryan, Daniel, Staats, Peter, Sumner, Jr., Robert
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US15/093,120
• Publication Number:
  US 20160227153A1
• Time in Patent Office: 670 Days
• Field of Search: 348 61
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 7/484 Transmitters
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04B 10/1149 Arrangements for indoor wireless networking of information
• CPC Class Codes: H04B 10/116 Visible light communication
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/455 Demodulation-circuits

Abstract

In one aspect, the present disclosure relates to a method for removing noise from a digital video sequence containing a modulated light signal emitted from a beacon light source. In some embodiments, the method includes electronically receiving, by an image sensor of a device, a digital video sequence of a scene, calculating noise from the digital video sequence, wherein the noise comprises information within the digital video sequence corresponding to the un-modulated illumination of the scene, reducing the noise from the digital video sequence to obtain an isolated digital video sequence of the modulated illumination of the scene, and demodulating the emitted light signal from the isolated digital video sequence.

Claims

1. A method comprising:

electronically receiving, by an image sensor of a device, a still image of a scene, wherein;

the scene comprises both modulated illumination emitted by a beacon light source and un-modulated illumination; and

the still image comprises a plurality of pixel values, each of the plurality of pixel values being for a given point in the still image;

calculating noise from the still image, wherein the noise comprises information within the still image corresponding to the un-modulated illumination of the scene;

removing the calculated noise from the still image to obtain an isolated digital image of the modulated illumination of the scene; and

demodulating the emitted light signal from the isolated digital image.

2. The method of claim 1, wherein removing the calculated noise comprises subtracting the plurality of pixel values corresponding to the un-modulated illumination of the scene from the plurality of pixel values corresponding to the modulated illumination of the scene.

3. The method of claim 1, wherein demodulating the emitted light signal from the isolated digital image comprises:

applying a blob detection algorithm to the isolated digital image, wherein a blob represents a region of interest identifiable as one of a plurality of beacon light sources, the blob detection algorithm including;

determining an average pixel value for the region of interest;

comparing the average pixel value to a threshold value; and

based on the comparison, storing a binary value for the region of interest in a processed image indicating where the plurality of beacon light sources are detected.

4. The method of claim 3, further comprising;

identifying a physical location of the device based on the processed image indicating where the plurality of beacon light sources are detected.

5. The method of claim 4, wherein the processed image is at a sampling rate equal to a frame rate of the image sensor.

6. The method of claim 1, wherein removing the calculated noise comprises:

applying phase correlation to detect scene motion between a plurality of frames, the plurality of frames including the still image;

upon detecting the scene motion between the plurality of frames, transforming the plurality of frames by shifting each of the plurality of frames to cause each frame to overlie a previous frame;

generating an averaged image by averaging a plurality of frames such that a banding effect due to the time-varying illumination of a plurality of beacon light sources is reduced to a constant value added on to all pixels of the averaged image; and

subtracting the averaged image from the still image to reveal the isolated digital image that is independent of the scene.

7. The method of claim 1, wherein calculating the noise comprises averaging the plurality of pixel values of each given point of the still image.

8. The method of claim 7, wherein the averaging is performed using a technique comprising one of calculating a simple numerical average, calculating a normalized average, calculating a Gaussian average, or calculating a difference between the still image and at least one prior digital image previously received electronically by the image sensor of the device.

9. The method of claim 1, wherein removing the calculated noise comprises subtracting the calculated noise from the still image to obtain the isolated digital image.

10. The method of claim 9, wherein demodulating the emitted light signal from the isolated digital image comprises taking the Fourier transform of row values along different columns of the isolated digital image.

11. An apparatus comprising:

an imaging sensor; and

a processor in communication with the imaging sensor configured to;

electronically receive a still image of a scene, wherein;

the scene comprises both modulated illumination emitted by a beacon light source and un-modulated illumination; and

the still image comprises a plurality of pixel values, each of the plurality of pixel values being for a given point in the still image;

calculate noise from the still image, wherein the noise comprises information within the still image corresponding to the un-modulated illumination of the scene;

remove the calculated noise from the still image to obtain an isolated digital image of the modulated illumination of the scene; and

demodulate the emitted light signal from the isolated digital image.

12. The apparatus of claim 11, wherein the processor is configured to remove the calculated noise using a technique comprising subtracting the plurality of pixel values corresponding to the un-modulated illumination of the scene from the plurality of pixel values corresponding to the modulated illumination of the scene.

13. The apparatus of claim 11, wherein demodulating the emitted light signal from the isolated digital image comprises:

applying a blob detection algorithm to the isolated digital image, wherein a blob represents a region of interest identifiable as one of a plurality of beacon light sources, the blob detection algorithm including;

determining an average pixel value for the region of interest;

comparing the average pixel value to a threshold value; and

based on the comparison, storing a binary value for the region of interest in a processed image indicating where the plurality of beacon light sources are detected.

14. The apparatus of claim 13, wherein the processor is configured to identify a physical location of the apparatus based on the processed image indicating where the plurality of beacon light sources are detected.

15. The apparatus of claim 14, wherein the processed image is at a sampling rate equal to a frame rate of the image sensor.

16. The apparatus of claim 11, wherein removing the calculated noise comprises:

applying phase correlation to detect scene motion between a plurality of frames, the plurality of frames including the still image;

upon detecting the scene motion between the plurality of frames, transforming the plurality of frames by shifting each of the plurality of frames to cause each frame to overlie a previous frame;

generating an averaged image by averaging a plurality of frames such that a banding effect due to the time-varying illumination of a plurality of beacon light sources is reduced to a constant value added on to all pixels of the averaged image; and

subtracting the averaged image from the still image to reveal the isolated digital image that is independent of the scene.

17. The apparatus of claim 11, wherein the processor is configured to perform the calculating of noise using a technique comprising averaging the plurality of pixel values of each given point of the still image.

18. The apparatus of claim 17, wherein the processor is configured to perform averaging using a technique comprising one of calculating a simple numerical average, calculating a normalized average, calculating a Gaussian average, or calculating a difference between the still image and at least one prior digital image previously received electronically by the imaging sensor.

19. The apparatus of claim 11, wherein removing the calculated noise comprises subtracting the calculated noise from the still image to obtain the isolated digital image.

20. The apparatus of claim 19, wherein demodulating the emitted light signal from the isolated digital image comprises taking the Fourier transform of row values along different columns of the isolated digital image.

21. A device comprising:

an image sensor;

a central processing unit (CPU) coupled to the image sensor; and

storage accessible to the CPU;

wherein the CPU is configured to operate the image sensor to;

electronically receive, by the image sensor of the device, a still image of a scene, wherein;

the scene comprises both modulated illumination emitted by a beacon light source and un-modulated illumination; and

the still image comprises a plurality of pixel values, each of the plurality of pixel values being for a given point in the still image;

calculate noise from the still image, wherein the noise comprises information within the still image corresponding to the un-modulated illumination of the scene;

remove the calculated noise from the still image to obtain an isolated digital image of the modulated illumination of the scene; and

demodulate the emitted light signal from the isolated digital image.

22. The device of claim 21, wherein:

the image sensor is a rolling shutter type camera; and

the demodulating of the emitted light signal from the isolated digital image comprises capturing data in response to characteristics of stripes in the still image from the rolling shutter camera caused by the modulated illumination.

23. The device of claim 22, wherein the CPU is further configured to:

correlate the captured data to a light source identifier associated with the beacon light source; and

calculate an estimate of a physical location of the device based on a location of the correlated beacon light source identifier.

24. The device of claim 23, wherein the location of the correlated beacon light source identifier is global positioning system (GPS) coordinates mapped into an indoor space.

25. The device of claim 22, wherein the CPU is further configured to:

transmit the captured data to a server; and

in response to transmitting the captured data, receive a calculation of a physical location of the device based on the captured data from the server.