Optical interference mitigation

• US 9,118,782 B1
• Filed: 09/19/2011
• Issued: 08/25/2015
• Est. Priority Date: 09/19/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Coley, Christopher, Shepelev, Petr G.
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Rahman, Mohammad J
• Application Number:
  US13/236,294
• Time in Patent Office: 1,436 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 9/00 Details of colour television systems

Abstract

Structured light generated by a projector may be recovered by a camera to determine three-dimensional data about a scene. The structured light signal acquired by the camera may be impaired due to interference from flickering sources of illumination such as lights, electronic displays, and so forth. This optical interference to an image acquisition system, such as a structured light system, is mitigated by modifying operation of a camera by applying a sampling coefficient to the camera'"'"'s frame rate or by presenting structured light frames during times non-coincident with the interference.

Claims

1. A system comprising:

a processor;

a projector coupled to the processor and configured to generate structured light at a projector frame rate;

a camera coupled to the processor and configured to capture at least a portion of the structured light after interaction with one or more objects;

memory, accessible by the processor; and

an interference mitigation module stored in the memory and executable by the processor, the interference mitigation module configured to;

detect interference by an emissive interference source to the structured light captured by the camera;

dynamically determine a temporal sampling coefficient “

k”

for at least partly reducing a level of the detected interference;

dynamically configure the camera to operate at a sample frame rate comprising a product of the temporal sampling coefficient “

k” and

the projector frame rate; and

apply a filter to frames acquired by the camera at the sample frame rate to remove at least a portion of individual frames acquired by the camera and affected by the detected interference.

2. The system of claim 1, wherein the detection of the interference comprises finding periodic variations of amplitude across a series of image frames.

3. The system of claim 1, wherein the detection of the interference comprises finding periodic recurring shapes across a series of image frames.

4. The system of claim 1, wherein the removed individual frames contain an interfering signal above a pre-determined amplitude threshold.

5. The system of claim 1, wherein:

the interference mitigation module is further configured to dynamically determine a frequency of the interference; and

the dynamically determining of the temporal sampling coefficient “

k”

is based at least in part on the frequency of the interference.

6. The system of claim 1, wherein the filter comprises one or more low-pass filters.

22. The system of claim 5, wherein the interference mitigation module determines the frequency of the interference by accessing a datastore that includes an interference frequency associated with the emissive interference source.

23. The system of claim 22, wherein the datastore is populated at least in part by detecting, by the camera, the emissive interference source and writing, by the processor, data associated with the emissive interference source to the datastore.

7. One or more non transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

dynamically determining a temporal sampling coefficient for mitigating interference in an image stream acquired by a camera capturing at least a portion of structured light generated by a structured light source;

dynamically configuring the camera to acquire the image stream at a sample frame rate comprising a product of the temporal sampling coefficient and a frame rate or flicker rate of the structured light source; and

applying a filter to one or more individual frames in the image stream to remove at least a portion of remaining instances of the interference.

8. The one or more non transitory computer-readable storage media of claim 7, wherein the temporal sampling coefficient is not equal to one.

9. The one or more non transitory computer-readable storage media of claim 7, the acts further comprising:

determining a source of the interference; and

accessing, from a datastore, an interference frequency of the source of the interference,

wherein the dynamically determining the temporal sampling coefficient is based at least in part on the interference frequency of the source of the interference.

10. The one or more non transitory computer-readable storage media of claim 7, the acts further comprising detecting the interference to the structured light captured by the camera by analyzing variations in intensity between a plurality of frames in the stream of frames.

11. The one or more non transitory computer-readable storage media of claim 10, wherein the interference comprises narrow-band interference having a frequency less than the frame rate or flicker rate of the structured light source.

12. A system comprising:

a processor;

a projector coupled to the processor and configured to generate structured light;

a camera coupled to the processor and configured to capture a series of image frames at a camera frame rate, the image frames containing at least a portion of the generated structured light after interaction with one or more objects;

memory, accessible by the processor and configured to store at least a portion of the series of image frames;

an interference mitigation module stored in the memory and configured to;

detect interference to the structured light in the series of image frames and intervals at which the interference occurs in the series of image frames; and

dynamically configure the projector to operate at a projector frame rate such that structured light is projected for capture by the camera at times non-coincident with the intervals at which the interference occurs.

13. The system of claim 12, wherein the interference mitigation module is further configured to configure the camera to operate such that the camera frame rate differs from the projector frame rate.

14. The system of claim 12, wherein the interference mitigation module is further configured to:

identify an object generating the interference; and

retrieve, from a datastore, the intervals of the object generating the interference at least partly in response to the identifying.

15. The system of claim 12, wherein the interference mitigation module is further configured to shift frame start timing of the projector such that the structured light is projected for capture by the camera at the times non-coincident with the intervals at which the interference occurs.

16. The system of claim 15, wherein the shifting comprises transposing frames containing structured light with frames containing non-structured light.

17. One or more non transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

determining intervals at which interference to a structured light signal occurs during imaging by a camera of the structured light signal; and

dynamically configuring a projector to operate at a projector frame rate such that structured light is presented at one or more times non-coincident with the intervals at which the interference occurs, the structured light being captured by the camera to create the structured light signal.

18. The one or more non transitory computer-readable storage media of claim 17, wherein the determining of the intervals comprises:

identifying an object generating the interference; and

retrieving, from a datastore, the intervals of the object generating the interference at least partly in response to the identifying.

19. The one or more non transitory computer-readable storage media of claim 17, the acts further comprising configuring the camera to operate at a frame rate greater than the projector frame rate of the projector generating the structured light.

20. The one or more non transitory computer-readable storage media of claim 17, the acts further comprising shifting a frame start time of the projector generating the structured light such that the structured light is presented at the one or more times non-coincident with the intervals at which the interference occurs.

21. The one or more non transitory computer-readable storage media of claim 17, wherein the shifting the frame start time comprises shifting the frame start time by an integer number of frames.