Rapid synchronized lighting and shuttering

• US 8,786,767 B2
• Filed: 11/02/2012
• Issued: 07/22/2014
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rihn, Bernard K., Dietz, Paul Henry, Keam, Nigel Stuart, Bathiche, Steven Nabil
• Primary Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US13/667,408
• Publication Number:
  US 20140125864A1
• Time in Patent Office: 627 Days
• Field of Search: 34820799-20711, 34821199-21114, 348/222.1, 3482261-2281, 348/239, 348294-324, 348335-340, 348/342, 348/344, 348370-371, 396 61- 62, 396155-206, 396322-340
• US Class Current: 348/371
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

This document describes various apparatuses and techniques for rapid synchronized lighting and shuttering. These apparatuses and techniques are capable of capturing two images, where one of the images integrates multiple exposures during which an image area is illuminated by a light source and another of the images integrates multiple exposures during which the image is not illuminated by the light source. The image area can be illuminated by rapidly flashing the image area with the light source and synchronizing a shutter to permit one image sensor to capture an image when the image area is illuminated and another image sensor to capture the image area when the image area is not illuminated. These two images can be captured concurrently or nearly concurrently, thereby reducing or eliminating motion artifacts. Further, these apparatuses and techniques may do so with slow and relatively low-cost cameras and relatively low computational costs.

Claims

1. A method comprising:

rapidly flashing at a flash rate at least twice as fast as an image capture rate of a first image sensor, or a second image sensor an object with a first light source, the object illuminated by a second light source in addition to the first light source;

synchronizing shuttering of image sensors to the rapid flashing, the synchronized shuttering exposing the first image sensor during the flashes and the second image sensor not during the flashes;

capturing, at the first image sensor, a first image of the object, the first image integrating multiple exposures during which the object is illuminated by the first light source during multiple, respective flashes and during which the object is also illuminated by the second light source;

capturing, at the second image sensor, a second image of the object, the second image integrating multiple exposures during which the object is not illuminated by the first light source but is illuminated by the second light source; and

comparing the first image and the second image effective to create a net image, the net image showing the object illuminated by the first light source but not illuminated by the second light source.

2. A method as recited in claim 1, wherein the rapidly flashing is not periodic and wherein an amount of time during which the second image sensor is exposed is greater than an amount of time during which the first image sensor is exposed.

3. A method as recited in claim 1, wherein the second light source is ambient light and the net image shows the object without being illuminated by the ambient light.

4. A method as recited in claim 1, wherein exposures of the first image sensor and the second image sensor are interleaved during a same period in which the first and second images are captured.

5. A method as recited in claim 1, wherein each of the image sensors is an electronic image sensor of a digital camera having an image recording speed that is at least twice as long as a speed of the rapid flashing.

6. A method as recited in claim 1, wherein the synchronized shuttering causes:

a first mechanical shutter to open prior to or during each flash of the rapid flashes and close prior to or during each flash of the rapid flashes, the first mechanical shutter interposed between the object and the first image sensor; and

a second mechanical shutter to open when or after each flash of the rapid flashes ceases and close prior to or when each flash of the rapid flashes begins, the second mechanical shutter interposed between the object and the second image sensor.

7. A method as recited in claim 1, wherein the synchronized shuttering rotates one or more ferro-electric polarization retarders placed between polarized light from the object and the first image sensor and the second image sensor effective to expose the first image sensor or the second image sensor but not both the first image sensor and the second image sensor.

8. An apparatus comprising:

multiple image sensors, each of the multiple image sensors having an image-capture rate;

a light source configured for flashing at a flash rate at least twice as fast as the image-capture rate;

a shutter configured for shuttering at a shuttering rate at least as fast as the flash rate; and

a controller configured for;

synchronizing flashing of the light source and shuttering of the shutter effective to enable a first of the multiple image sensors to capture a first image integrating multiple exposures during which the light source is flashed and a second of the multiple image sensors to capture a second image integrating multiple exposures during which the light source is not flashed; and

providing the first image and the second image, the first image and the second image usable to determine a net image in which one or more additional light sources other than the light source are excluded.

9. An apparatus as recited in claim 8, wherein the apparatus includes a second light source of the one or more additional light sources, the second light source illuminating an object captured by the first and second images and excludable from the net image.

10. An apparatus as recited in claim 8, wherein the light source is an LED or laser diode and the flash rate of the LED or laser diode is at least 60 Hz.

11. An apparatus as recited in claim 8, wherein:

the multiple image sensors have image-capture rates being 60 image captures per second or slower;

the multiple exposures integrated into the first image include four exposures;

the shuttering rate of the shutter is greater than once per 500 microseconds; and

the flash rate is greater than one flash per 500 microseconds.

12. An apparatus as recited in claim 8, wherein the shutter includes a beam splitter, a polarizer, and a ferro-electric polarization retarder.

13. An apparatus as recited in claim 12, wherein the beam splitter is a silvered mirror or a dual brightness enhancement film (DBEF) sheet.

14. An apparatus as recited in claim 8, wherein the controller is computer hardware operating without computer-readable instructions or a processor.

15. An apparatus as recited in claim 8, further comprising:

a second light source configured for flashing at the flash rate or faster; and

a second shutter configured for shuttering at the shuttering rate or faster, and wherein;

the multiple image sensors include a first pair of image sensors and a second pair of image sensors, the first pair of image sensors including the first image sensor and the second image sensor; and

the controller is further configured for;

synchronizing flashing of the second light source and shuttering of the second shutter effective to enable the second pair of image sensors to capture third and fourth images integrating multiple exposures during which the second light source is flashed; and

providing the third and fourth images, the first image, and the second image usable to determine placement of a moving object in three dimensions based on a first shadow of the moving object created by the light source and a second shadow of the moving object created by the second light source.

16. One or more computer-readable storage memory having computer-executable instructions thereon that, when executed by one or more processors, perform acts comprising:

rapidly flashing, at least four times faster than an image-sensor frame rate, an object with a first light source, the object illuminated by ambient light in addition to the first light source;

synchronizing shuttering of image sensors to the rapid flashing, the synchronized shuttering exposing a first image sensor during the flashes and a second image sensor not during the flashes;

capturing, at the first image sensor and at the image-sensor frame rate, a first image of the object, the first image integrating two or more exposures during which the object is illuminated by the first light source during multiple, respective flashes and during which the object is also illuminated by the ambient light;

capturing, at the second image sensor and at the image-sensor frame rate, a second image of the object, the second image integrating two or more exposures during which the object is not illuminated by the first light source but is illuminated by the ambient light; and

providing the first image and the second image effective to enable creation of a net image, the net image showing the object illuminated by the first light source but not illuminated by the ambient light.

17. One or more computer-readable storage memory having the computer-executable instructions as recited in claim 16, the computer-executable instructions further executable to perform an act comprising creating the net image by subtracting the second image from the first image.

18. One or more computer-readable storage memory having the computer-executable instructions as recited in claim 16, the computer-executable instructions further executable to perform an act comprising band-pass filtering the first image at a frequency band of the first light source.

19. One or more computer-readable storage memory having the computer-executable instructions as recited in claim 16, wherein the object is a hand, finger, or stylus, and wherein the net image is effective to enable determination of a gesture performed by the hand, the finger, or the stylus.

20. One or more computer-readable storage memory having the computer-executable instructions as recited in claim 19, wherein the ambient light is generated by a computer screen over which the hand, the finger, or the stylus performs the gesture.