Method and apparatus for automatic visual event detection

• US 8,243,986 B2
• Filed: 05/26/2005
• Issued: 08/14/2012
• Est. Priority Date: 06/09/2004
• Status: Active Grant

• Current Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Original Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Inventors: Silver, William M., Phillips, Brian S.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US11/138,025
• Publication Number:
  US 20050276461A1
• Time in Patent Office: 2,637 Days
• Field of Search: 382103-105, 382/107, 382/153, 382/291, 348143-160, 348169-172
• US Class Current: 382/103
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Disclosed are methods and apparatus for automatic visual detection of events, for recording images of those events and retrieving them for display and human or automated analysis, and for sending synchronized signals to external equipment when events are detected. An event corresponds to a specific condition, among some time-varying conditions within the field of view of an imaging device, that can be detected by visual means based on capturing and analyzing digital images of a two-dimensional field of view in which the event may occur. Events may correspond to rare, short duration mechanical failures for which obtaining images for analysis is desirable. Events are detected by considering evidence obtained from an analysis of multiple images of the field of view, during which time moving mechanical components can be seen from multiple viewing perspectives.

Claims

1. A method for detection and reporting of a plurality of events, comprising:

determining, for each event of the plurality of events, a corresponding mark time that is a function of a time at which the event occurs, wherein determining the corresponding mark time occurs at a corresponding decision point that differs from the corresponding mark time by a corresponding decision delay; and

the decision delays corresponding to the plurality of events are variable;

using an output signaler to produce, for each event of the plurality of events, a corresponding signal at a corresponding report time that follows the corresponding mark time by a delay interval; and

using, for each event of the plurality of events, the corresponding report time of the corresponding signal to indicate the corresponding mark time.

6. The method of claim 1, wherein each mark time is proportional to a time at which the corresponding event occurs.

7. The method of claim 1, wherein each mark time is an encoder count that is a function of a time at which the corresponding event occurs.

8. The method of claim 1, wherein each event of the plurality of events comprises a flow event, the flow event comprising motion of an object across a mark point, the corresponding mark time responsive to a time at which the object crosses the mark point.

9. The method of claim 1, wherein each event of the plurality of events comprises a stroke event, the stroke event comprising motion of an object advancing to and retreating from an apex, the corresponding mark time responsive to a time at which the object reaches the apex.

10. The method of claim 1, further comprising selecting an event type from a plurality of event types;

and wherein determining the corresponding mark time is responsive to the event type.

11. The method of claim 10, wherein the plurality of event types comprises a flow event type and a stroke event type.

12. The method of claim 10, wherein selecting the event type is responsive to a human-machine interface.

13. The method of claim 1, wherein the delay interval is selected responsive to a human-machine interface.

22. The method of claim 1, further comprising:

providing relative motion between a set of objects and a sensor, the set of objects comprising at least one object; and

providing a mark point that is in a fixed position relative to the sensor; and

wherein

each event of the plurality of events comprises a crossing of the mark point by an object from the set of objects.

23. The method of claim 22, further comprising

inputting an encoding signal responsive to the relative motion, from which can be obtained at desired times a corresponding encoder count indicating a location of an object from the set of objects relative to the sensor;

and wherein

each mark time comprises an encoder count.

24. The method of claim 22, further comprising

operating on at least a portion of the set of objects with an actuator at an actuator location;

and wherein

a location of an object from the set of objects at the report time corresponds to the actuator location.

25. The method of claim 22, wherein determining each corresponding mark time further comprises:

using the sensor to capture a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view;

choosing a plurality of active frames from the plurality of frames;

obtaining a plurality of capture times corresponding to the plurality of active frames,

each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured;

performing an analysis of the plurality of active frames to obtain a plurality of location values, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the time at which the corresponding active frame was captured; and

using the plurality of location values and the plurality of capture times to determine the mark time.

26. The method of claim 25, wherein choosing the plurality of active frames from the plurality of frames further comprises:

performing an analysis of at least a portion of the plurality of frames to obtain evidence that an object from the set of objects is located in the field of view; and

choosing as the plurality of active frames at least a portion of the plurality of frames where the evidence is sufficient.

27. The method of claim 25, wherein determining the mark time further comprises fitting a curve to at least three capture times and corresponding location values.

28. The method of claim 27, wherein the curve comprises a line.

29. The method of claim 1, further comprising using a first-in first-out buffer to hold information needed for producing the signal at the report time.

30. The method of claim 1, further comprising

providing relative motion between a set of objects and a sensor, the set of objects comprising at least one object, the relative motion comprising motion of advance towards and retreat from an apex;

and wherein

each event of the plurality of events comprises a reaching of the apex by an object from the set of objects.

31. The method of claim 1, further comprising adjusting the delay interval so that the report time corresponds to a desired downstream location of the object.

32. The method of claim 1, wherein the signal comprises a pulse, and wherein an edge of the pulse corresponds to the report time.

2. A system for detecting and reporting of a plurality of events, comprising:

a first analysis process that determines, for each event of the plurality of events, a corresponding mark time that is a function of a time at which the event occurs, wherein

the first analysis process determines the corresponding mark time at a corresponding decision point that differs from the corresponding mark time by a corresponding decision delay; and

the decision delays corresponding to the plurality of events are substantially variable; and

an output signaler that produces, for each event of the plurality of events, a corresponding signal at a corresponding report time that follows the corresponding mark time by a delay interval.

14. The system of claim 2, wherein each mark time is proportional to a time at which the corresponding event occurs.

15. The system of claim 2, wherein each mark time is an encoder count that is a function of a time at which the corresponding event occurs.

16. The system of claim 2, wherein each event of the plurality of events comprises a flow event, the flow event comprising motion of an object across a mark point, the corresponding mark time responsive to a time at which the object crosses the mark point.

17. The system of claim 2, wherein each event of the plurality of events comprises a stroke event, the stroke event comprising motion of an object advancing to and retreating from an apex, the corresponding mark time responsive to a time at which the object reaches the apex.

18. The system of claim 2, further comprising a selection process that selects an event type from a plurality of event types;

and wherein the analysis process determines the corresponding mark time using the event type.

19. The system of claim 18, wherein the plurality of event types comprises a flow event type and a stroke event type.

20. The system of claim 18, further comprising a human-machine interface that is used to select the event type.

21. The system of claim 2, further comprising a human-machine interface that is used to select the delay interval.

33. The system of claim 2, further comprising:

a sensor;

a motion process that provides relative motion between a set of objects and the sensor, the set of objects comprising at least one object; and

a mark point that is in a fixed position relative to the sensor; and

wherein

each event of the plurality of events comprises a crossing of the mark point by an object from the set of objects.

34. The system of claim 33, further comprising

an encoder responsive to the relative motion, from which can be obtained at desired times a corresponding encoder count indicating a location of an object from the set of objects relative to the sensor;

and wherein

each mark time comprises an encoder count.

35. The system of claim 33, further comprising

an actuator that operates on at least a portion of the set of objects at an actuator location;

and wherein

a location of an object from the set of objects at the report time corresponds to the actuator location.

36. The system of claim 33, further comprising:

a capture process that captures a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view;

a selection process that chooses a plurality of active frames from the plurality of frames;

a timing process that obtains a plurality of capture times corresponding to the plurality of active frames, each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured; and

a second analysis process that analyzes the plurality of active frames to compute a plurality of location values, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the capture time of the corresponding active frame; and

wherein

the first analysis process determines the mark time using the plurality of location values and the plurality of capture times.

37. The system of claim 36, wherein the selection process analyzes at least a portion of the plurality of frames to obtain evidence that an object from the set of objects is located in the field of view and chooses as the plurality of active frames at least a portion of the plurality of frames where the evidence is sufficient.

38. The system of claim 36, wherein the first analysis process further comprises fitting a curve to at least three capture times and corresponding location values.

39. The system of claim 38, wherein the curve comprises a line.

40. The system of claim 2, further comprising a first-in first-out buffer to hold information needed for producing the signal at the report time.

41. The system of claim 2, further comprising

a motion process that provides relative motion between a set of objects and a sensor, the set of objects comprising at least one object, the relative motion comprising motion of advance towards and retreat from an apex;

and wherein

each event of the plurality of events comprises a reaching of the apex by an object from the set of objects.

42. The system of claim 2, further comprising a controller that adjusts the delay interval so that the report time corresponds to a desired downstream location of the object.

43. The system of claim 2, wherein the signal comprises a pulse, and wherein an edge of the pulse corresponds to the report time.

44. The system of claim 2, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the system further adapted to:

receive a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view, the image having more than two pixel values;

select a plurality of active frames from the plurality of frames;

receive a plurality of capture times corresponding to the plurality of active frames, each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured; and

compute a plurality of location values based on an analysis of the more than two pixel values of the plurality of active frames, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the capture time of the corresponding active frame; and

wherein

the first analysis process determines the mark time using the plurality of location values and the plurality of capture times.

45. The system of claim 44, wherein the capture times are a function of the time the image is captured by an imager with a global shutter.

46. The system of claim 44, wherein the plurality of location values is at least two location values.

47. The system of claim 44, wherein the plurality of location values is at least three location values.

48. The system of claim 2, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, the set of objects and the sensor being in relative motion at an unknown relative velocity.

3. An article of manufacture including a tangible computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

computing for each event of a plurality of events, a corresponding mark time that is a function of a time at which the event occurs, wherein computing the corresponding mark time occurs at a corresponding decision point,

the decision point that differs from the corresponding mark time by a corresponding decision delay; and

the decision delays corresponding to the plurality of events are substantially variable; and

generating for each event of the plurality of events, a corresponding signal at a corresponding report time that follows the corresponding mark time by a delay interval.

49. The article of claim 3, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the tangible computer-readable medium having further instructions stored thereon, that if executed by a computing device, cause the computing device to perform operations further comprising:

receive a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view, the image having more than two pixel values;

select a plurality of active frames from the plurality of frames;

receive a plurality of capture times corresponding to the plurality of active frames, each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured; and

compute a plurality of location values based on an analysis of the more than two pixel values of the plurality of active frames, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the capture time of the corresponding active frame; and

wherein

the first analysis process determines the mark time using the plurality of location values and the plurality of capture times.

50. The article of claim 49, wherein the plurality of location values is at least three location values.

51. The article of claim 50, wherein the first analysis process further comprises fitting a curve to at least three capture times and corresponding location values.

52. The article of claim 51, wherein the curve comprises a line.

53. The article of claim 49, wherein the selecting analyzes at least a portion of the plurality of frames to obtain evidence that an object from the set of objects is located in the field of view and chooses as the plurality of active frames at least a portion of the plurality of frames where the evidence is sufficient.

54. The article of claim 49, wherein the capture times are a function of the time the image is captured by an imager with a global shutter.

55. The article of claim 49, wherein the plurality of location values is at least two location values.

56. The article of claim 3, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the tangible computer-readable medium having further instructions stored thereon, that if executed by a computing device, cause the computing device to perform operations further comprising:

receiving an encoder count indicating a location of an object from the set of objects relative to the sensor; and

wherein each mark time comprises an encoder count.

57. The article of claim 3, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, and wherein the set of objects and the sensor being in relative motion at an unknown relative velocity.

58. The article of claim 3, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, and wherein the event comprises reaching of an apex by an object from a set of objects, an object of set of objects and the sensor being in relative motion as the object advances towards and retreats from the apex.

59. The article of claim 3, the tangible computer-readable medium having further instructions stored thereon, that if executed by a computing device, cause the computing device to perform operations further comprising:

holding in a first-in first-out buffer information needed for producing the signal at the report time.

60. The article of claim 3, further comprising a controller that adjusts the delay interval so that the report time corresponds to a desired downstream location of the object.

61. The article of claim 3, wherein the signal comprises a pulse, and wherein an edge of the pulse corresponds to the report time.

4. A method for detecting and reporting of a plurality of events, comprising:

computing, by a processor, for each event of the plurality of events, a corresponding mark time that is a function of a time at which the event occurs, wherein determining the corresponding mark time occurs at a corresponding decision point that differs from the corresponding mark time by a corresponding decision delay;

the decision delays corresponding to the plurality of events are variable; and

generating, by a processor, for events of the plurality of events a signal at a report time that follows the corresponding mark time by a delay interval;

the report time of the signal indicating the corresponding mark time.

62. The method of claim 4, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the method further comprising:

receiving, by a processor, a plurality of frames, each frame in the plurality of frames comprising a image of a two-dimensional field of view, the image having more than two pixel values;

selecting, by a processor, a plurality of active frames from the plurality of frames;

receiving, by a processor, a plurality of capture times corresponding to the plurality of active frames, each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured; and

computing, by a processor, a plurality of location values based on an analysis of the more than two pixel values of the plurality of active frames, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the capture time of the corresponding active frame; and

wherein the first analysis process determines the mark time using the plurality of location values and the plurality of capture times.

63. The method of claim 62, wherein the capture times are a function of the time the image is captured by an imager with a global shutter.

64. The method of claim 62, wherein the plurality of location values is at least two location values.

65. The method of claim 62, wherein the plurality of location values is at least three location values.

66. The method of claim 4, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, the set of objects and the sensor being in relative motion at an unknown relative velocity.

67. The method of claim 4, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the method further comprising:

receiving, by a processor, an encoder count indicating a location of an object from the set of objects relative to the sensor; and

wherein each mark time comprises an encoder count.

5. A system for detecting and reporting a plurality of events, the system comprising:

a data processing device programmed to;

compute for each event of the plurality of events, a corresponding mark time that is a function of a time at which the event occurs, wherein determining the corresponding mark time occurs at a corresponding decision point that differs from the corresponding mark time by a corresponding decision delay;

the decision delays corresponding to the plurality of events are variable; and

generate for events of the plurality of events a signal at a report time that follows the corresponding mark time by a delay interval;

the report time of the signal indicating the corresponding mark time.

68. The system of claim 5, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the method, the data processing device programmed to further perform operations comprising:

receive a plurality of frames, each frame in the plurality of frames comprising an image of a two-dimensional field of view, the image having more than two pixel values;

select a plurality of active frames from the plurality of frames;

receive a plurality of capture times corresponding to the plurality of active frames, each capture time of the plurality of capture times being a function of a time at which the corresponding active frame was captured; and

compute a plurality of location values based on an analysis of the more than two pixel values of the plurality of active frames, each location value of the plurality of location values responsive to a position relative to the mark point of an object from the set of objects at the capture time of the corresponding active frame; and

wherein

the first analysis process determines the mark time using the plurality of location values and the plurality of capture times.

69. The system of claim 68, wherein the plurality of location values is at least three location values.

70. The system of claim 69, wherein the first analysis process further comprises fitting a curve to at least three capture times and corresponding location values.

71. The system of claim 70, wherein the curve comprises a line.

72. The system of claim 68, wherein the capture times are a function of the time the image is captured by an imager with a global shutter.

73. The system of claim 68, wherein the plurality of location values is at least two location values.

74. The system of claim 68, wherein the selecting analyzes at least a portion of the plurality of frames to obtain evidence that an object from the set of objects is located in the field of view and chooses as the plurality of active frames at least a portion of the plurality of frames where the evidence is sufficient.

75. The system of claim 5, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, the set of objects and the sensor being in relative motion at an unknown relative velocity.

76. The system of claim 5, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the set of objects and the sensor being in relative motion, the data processing device programmed to further perform operations comprising:

receiving an encoder count indicating a location of an object from the set of objects relative to the sensor; and

wherein each mark time comprises an encoder count.

77. The system of claim 5, wherein the event comprises crossing a mark point by an object from a set of objects, the mark point being in a fixed position relative to a sensor, wherein the event comprises reaching of an apex by an object from a set of objects, an object of set of objects and the sensor being in relative motion as the object advances towards and retreats from the apex.

78. The system of claim 5, further comprising a first-in first-out buffer to hold information needed for producing the signal at the report time.

79. The system of claim 5, further comprising a controller that adjusts the delay interval so that the report time corresponds to a desired downstream location of the object.

80. The system of claim 5, wherein the signal comprises a pulse, and wherein an edge of the pulse corresponds to the report time.