Synchronizing active illumination cameras

• US 10,462,452 B2
• Filed: 03/16/2016
• Issued: 10/29/2019
• Est. Priority Date: 03/16/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kashyap, Abhinav
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Tekle, Daniel T
• Application Number:
  US15/071,246
• Publication Number:
  US 20170272731A1
• Time in Patent Office: 1,322 Days
• Field of Search: None
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/003 Transmission of data between radar, sonar or lidar systems and remote stations
• CPC Class Codes: G01S 7/484 Transmitters
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

Apparatus for controlling a plurality of active illumination cameras to operate in a frequency division multiplexed operating mode to acquire images of scenes that the cameras image.

Claims

1. A system for controlling operation of a plurality of active illumination time of flight (TOF) range cameras, the system comprising:

a hub configured to communicate with mobile communication devices and comprising a database of active illumination TOF range cameras subscribed to the system; and

a set of computer executable instructions comprised in each subscriber TOF range camera that configures the camera to communicate with the hub when located in a same imaging neighbourhood with other subscriber TOF range cameras to establish a frequency division multiplexing (FDM) imaging mode of operation in which each of the subscriber TOF range cameras is assigned a different FDM imaging frequency that the TOF range camera uses to determine a modulation period for light that the camera transmits to illuminate a scene that the camera images and a modulation period for modulating sensitivity of the TOF range camera to register light reflected by features in the scene from the transmitted modulated light.

2. The system according to claim 1 wherein the subscriber TOF range cameras are configured to establish a wireless local area network (WLAN) over which the cameras in the imaging neighbourhood communicate to implement the FDM imaging mode.

3. The system according to claim 2 wherein each TOF range camera is configured to perform a radio scan after being turned on in the imaging neighbourhood to detect a radio beacon broadcast by another subscriber TOF range camera in the imagining neighbourhood, the radio beacon comprising a service set identifier (SSID) for the WLAN.

4. The system according to claim 3 wherein if the TOF range camera does not detect the radio beacon, the TOF range camera is configured to communicate with the hub and request an SSID for establishing the WLAN, and an access code that another TOF range camera may present to be granted access to the WLAN.

5. The system according to claim 4 wherein the hub is configured to respond to the request and communicate an SSID and access code to the TOF range camera.

6. The system according to claim 5 wherein, based at least in part on receiving the SSID and access code, the TOF range camera is configured to:

broadcast a radio beacon comprising the SSID;

assume a role as a master camera of the WLAN that accepts as members of the WLAN in the role of slave cameras, other subscriber TOF range cameras that detect the beacon and transmit a request comprising the access code to join the WLAN; and

administer the FDM imaging mode of operation by allocating unique FDM imaging frequencies to slave cameras, which the slave cameras use to determine modulation periods for modulating light that the cameras transmit and modulating sensitivity of the slave cameras to register light.

7. The system according to claim 6 wherein the master camera assigns each slave camera at least one or any combination of more than one of a light pulse width, light pulse intensity, and duration of a pulse train comprising light pulses the camera transmits.

8. The system according to claim 7 wherein the master camera is operable to receive an operating profile of the slave camera that is transmitted to the master camera over the WLAN, and based on the operating profile customize at least one or any combination of more than one of the FDM imaging frequency, light pulse width, light pulse intensity and pulse train duration.

9. The system according to claim 8 wherein the operating profile comprises at least one or any combination of more than one of:

an optical parameter that characterizes the slave camera, an electrical parameter that characterizes the slave camera, an operating temperature of the slave camera, power reserves of the camera, and distance from the slave camera of a scene to be range imaged by the slave TOF camera.

10. The system according to claim 3 wherein if the TOF range camera detects the radio beacon, the TOF range camera is configured to communicate with the hub to:

request and receive the access code for the WLAN from the hub;

after receiving the access code submit a request to the other subscriber camera acting as a master of the WLAN, to join the WLAN as a slave to the master; and

after joining the WLAN receive a unique FDM imaging frequency at which to operate to acquire range images.

11. The system according to claim 1 wherein each subscriber TOF range camera is configured to communicate with the hub when the subscriber TOF range camera is turned on, and transmit to the hub, data that may be used to determine a geolocation of the subscriber TOF range camera.

12. The system according to claim 11 wherein the hub is configured to process the geolocation data it receives from subscriber TOF range cameras to determine when two or more of the TOF range cameras are in a same imaging neighborhood in which the cameras may interfere with each other'"'"'s operation.

13. The system according to claim 12 wherein subsequent to determining that two or more of the subscriber TOF range cameras are located in a same imaging neighbourhood, the hub is configured to communicate with the cameras to allocate FDM imaging frequencies to the subscriber TOF range cameras in the neighbourhood so that the subscriber TOF range cameras may operate in the FDM imaging mode.

14. The system according to claim 1 wherein the active illumination TOF range cameras comprise gated TOF (GT-TOF) range cameras and wherein a GT-TOF range camera of the plurality of GT-TOF cameras assigned a lower FDM imaging frequency transmits light pulses to illuminate a scene the camera images having a larger pulse width than light pulses that a GT-TOF range camera of the plurality of GT-TOF range cameras assigned a higher FDM imaging frequency transmits to illuminate a scene that the GT-TOF range camera images.

15. The system according to claim 1 wherein the plurality of active illumination TOF range cameras comprises at least one gated TOF range camera and/or at least one continuous wave time of flight (CW-TOF) range camera.

16. The system according to claim 1 wherein the significands of the numerical values of the modulation periods expressed in scientific notation are relatively prime numbers.

17. A method of imaging a scene using a plurality of active illumination TOF range cameras, the method comprising:

establishing a wireless local area network (WLAN) over which the plurality of TOF range cameras communicate; and

appointing one of the cameras to transmit messages to other of the TOF range cameras assigning them different FDM imaging frequencies of operation at which to operate to acquire ranges images of scenes, wherein the frequency of operation of each TOF range camera of the plurality of TOF range cameras is established based on receiving an assigned frequency of operation from at least one other TOF range camera in the imaging neighborhood or assigning different frequencies of operation to the other TOF range cameras in the imaging neighborhood.

18. An active illumination time of flight (TOF) range camera operable to determine distances to features in a scene, the TOF camera comprising:

a light source configured to transmit temporally modulated light having a modulation period to illuminate the scene;

a photosensor comprising at least one pixel configured to register amounts of light reflected from the transmitted modulated light by a feature in the scene that is imaged on a pixel of the at least one pixel; and

a controller configured to;

establish an FDM imaging frequency of operation for the TOF range camera based on a presence of at least one other active illuminating camera in an imaging neighbourhood of the TOF range camera whose operation may interfere with operation of the TOF range camera, wherein the FDM imaging frequency of operation is established based on receiving an assigned FDM imaging frequency of operation from at least one other active illuminating camera in the imaging neighborhood or assigning different frequencies of operation to the TOF range camera and the at least one other active illuminating camera in the imaging neighborhood;

use the established FDM imaging frequency to determine a modulation period for the temporally modulated light that the TOF range camera transmits, and modulate sensitivity of the at least one pixel to register light reflected by features in the scene from the transmitted modulated light; and

determine a distance to the feature imaged on the pixel of the at least one pixel based on an amount of light that the pixel registers during modulation of the sensitivity of the at least one pixel.

19. The active illumination TOF range camera according to claim 18 wherein the controller is configured to cooperate in establishing a wireless local area network (WLAN) over which to communicate with the at least one other active illumination TOF range camera and wherein the TOF range camera is configured to establish the frequency of operation based on communications via the WLAN with the at least one other active illumination TOF range camera.

20. The active illumination TOF range camera according to claim 18 wherein the controller is configured to establish the frequency of operation by requesting and receiving the frequency from a hub.