Method and apparatus for automatic visual event detection

• US 8,249,297 B2
• Filed: 05/26/2005
• Issued: 08/21/2012
• Est. Priority Date: 06/09/2004
• Status: Active Grant

• Current Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Original Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Inventors: Silver, William M.
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US11/138,033
• Publication Number:
  US 20050276462A1
• Time in Patent Office: 2,644 Days
• Field of Search: 382/103, 382/107, 382/141, 382/154, 382/173, 348/88, 348/92, 348/125, 348/135, 348/143, 348/155, 348/157, 348/222.1, 707 1- 3, 715/804, 715/810, 717/109, 717/112, 717/113
• US Class Current: 382/103
• CPC Class Codes: G01N 2021/8887 based on image processing techniques
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

Disclosed are methods and apparatus for automatic visual detection of events, for recording images of those events and retrieving them for display and human or automated analysis, and for sending synchronized signals to external equipment when events are detected. An event corresponds to a specific condition, among some time-varying conditions within the field of view of an imaging device, that can be detected by visual means based on capturing and analyzing digital images of a two-dimensional field of view in which the event may occur. Events may correspond to rare, short duration mechanical failures for which obtaining images for analysis is desirable. Events are detected by considering evidence obtained from an analysis of multiple images of the field of view, during which time moving mechanical components can be seen from multiple viewing perspectives.

Claims

1. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs, the event comprising a motion of a predetermined object;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.

2. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights;

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred; and

producing a signal that is synchronized with a time at which the event occurs.

3. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred, the event analysis comprising a text string representing an expression in a syntax substantially similar to the syntax of a conventional programming language.

4. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs, the field of view comprising no more than about 40,000 pixels;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.

5. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights;

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred; and

wherein the steps of the method are performed at a rate of not less than two hundred frames per second.

6. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting, responsive to an activity analysis of the plurality of event detection weights, a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights, the activity analysis comprising

entering an active state when the visual analysis computes an event detection weight that indicates sufficient evidence that the event is occurring; and

exiting the active state when the visual analysis computes a plurality of event detection weights that each indicate insufficient evidence that the event is occurring; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.

7. A method for automatic visual detection of an event, comprising:

using a human-machine interface to specify the event, the human-machine interface comprising a logic view;

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.

8. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs, the event comprising a motion of a predetermined object;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred.

9. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights;

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred; and

an output process that produces a signal that is synchronized with a time at which the event occurs.

10. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred, the event analysis process comprising a text string representing an expression in a syntax substantially similar to the syntax of a conventional programming language.

11. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs, the capture process comprising an imager comprising no more than about 40,000 pixels;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred.

12. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights;

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred; and

wherein the capture process, the visual analysis process, the selection process, and the event analysis process operate at a rate of not less than two hundred frames per second.

13. A system for automatic visual detection of an event, comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

an activity analysis process that selects, responsive to the plurality of event detection weights, a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights, the activity analysis process comprising an active state that is entered when the visual analysis process computes an event detection weight that indicates sufficient evidence that the event is occurring, and that is exited when the visual analysis process computes a plurality of event detection weights that each indicate insufficient evidence that the event is occurring; and

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred.

14. A system for automatic visual detection of an event, comprising:

a human-machine interface that specifies the event, the human-machine interface comprising a logic view;

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

a visual analysis process adapted to be carried out by at least one of a collection of digital hardware elements and a collection of computer software instructions residing on a non-transitory computer-readable medium that computes, responsive to the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

a selection process that selects a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

an event analysis process that determines, responsive to the subset of the plurality of event detection weights, whether the event has occurred.

15. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs, wherein each frame comprises intensity values from a particular integration time;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.

16. The method of claim 15 wherein the plurality of frames are captured using an imager with a global shutter.

17. A method for automatic visual detection of an event, comprising:

capturing a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view in which the event occurs;

computing, using a processor, responsive to a visual analysis of the plurality of frames, a plurality of event detection weights, each event detection weight of the plurality of event detection weights corresponding respectively to each frame of the plurality of frames and comprising evidence that the event is occurring in the field of view;

selecting, responsive to an activity analysis of the plurality of event detection weights, a plurality of event frames from the plurality of frames, the plurality of event frames corresponding to a subset of the plurality of event detection weights, wherein only those frames in which there is sufficient evidence that the event has occurred are selected as a result of the activity analysis, the activity analysis comprising,

entering an active state when the visual analysis computes an event detection weight that indicates sufficient evidence that the event is occurring; and

exiting the active state when the visual analysis computes a plurality of event detection weights that each indicate insufficient evidence that the event is occurring; and

determining, responsive to an event analysis of the subset of the plurality of event detection weights, whether the event has occurred.