Apparatus and method for recognizing motion by using an event-based vision sensor

• US 9,389,693 B2
• Filed: 10/12/2012
• Issued: 07/12/2016
• Est. Priority Date: 10/14/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jun-Haeng, Ryu, Hyun-Surk, Park, Keun-Joo
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US14/351,806
• Publication Number:
  US 20140320403A1
• Time in Patent Office: 1,369 Days
• Field of Search: 345/104, 345156-176, 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 25/47 Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data

Abstract

An apparatus and method for recognizing motion by using an event-based vision sensor is provided. An apparatus for recognizing motion using an event-based vision sensor includes: a vision sensor to sense a movement-occurring part and output events; a movement type determiner configured to determine a type of movement using a frequency of occurrence of the events outputted through the vision sensor; a first motion determiner configured to track a movement trajectory of the movement-occurring part and determine a motion pattern based on the movement trajectory in response to a result of the movement type determination unit indicating a small movement; a second motion determiner configured to determine a direction of movement direction in which an object moves based on the events in response to a result of the movement type determination indicating a large movement; and a motion controller configured to output a control instruction to control a device.

Claims

1. An apparatus for recognizing motion using an event-based vision sensor, the apparatus comprising:

a vision sensor configured to sense a movement-occurring part and output events;

a movement type determiner configured to determine a type of movement using a frequency of occurrence of the events outputted through the vision sensor;

a first motion determiner configured to track a movement trajectory of the movement-occurring part and determine a motion pattern based on the tracked movement trajectory in response to a result of the movement type determiner indicating a small movement;

a second motion determiner configured to determine a direction of movement in which an object moves based on the output events in response to a result of the movement type determination indicating a large movement; and

a motion controller configured to output a control instruction to control a device based on the motion pattern determined by the first motion determiner or the movement direction determined by the second motion determiner.

2. The apparatus of claim 1, wherein the movement type determiner is configured to calculate an occurrence frequency of the events outputted through the vision sensor and compares the occurrence frequency to a predetermined threshold value to determine the occurrence frequency to be a small movement in which a portion of an object moves in response to the occurrence frequency being less than the threshold value, and determine the occurrence frequency to be a large movement in which a whole of the object moves in response to the occurrence frequency being greater than or equal to the threshold value.

3. The apparatus of claim 1, wherein the first motion determiner comprises:

spatiotemporal correlators which correspond to a plurality of predetermined receptive fields of the vision sensor that receive inputs of events outputted from the predetermined receptive fields and calculate a spatiotemporal correlation for each of the predetermined receptive fields;

a movement trajectory tracker configured to track a movement trajectory of the movement-occurring part based on a fluctuation in a spatiotemporal correlation of the predetermined receptive fields; and

a motion pattern determiner configured to determine a motion pattern of the movement-occurring part from the tracked movement trajectory.

4. The apparatus of claim 3, wherein the receptive fields correspond to divided areas of the vision sensor and overlap an area of another neighboring receptive field.

5. The apparatus of claim 3, wherein the movement trajectory tracker is configured to generate a cluster by grouping spatiotemporal correlators having a high spatiotemporal correlation from among the spatiotemporal correlators, sets a predetermined area including the cluster to be a clustering area, calculates a center of the clustering area to be a location of a moving object, and tracks a movement trajectory in conjunction with the calculated location of the moving object.

6. The apparatus of claim 5, wherein the movement trajectory tracker re-calculates a center of a previous clustering area based on an internal state value of spatiotemporal correlators in the previous clustering area, calculates the re-calculated center of the previous clustering area to be a location of the moving object, and tracks a movement trajectory in conjunction with the calculated location of the moving object in response to spatiotemporal correlators having a high spatiotemporal correlation being absent among the spatiotemporal correlators.

7. The apparatus of claim 1, wherein the second motion determiner comprises:

a first direction detection filter configured to receive an input of events outputted from a plurality of predetermined receptive fields of the vision sensor and detect a movement direction in which the object moves for the plurality of predetermined receptive fields; and

a second direction detection filter configured to determine a final movement direction of the object based on the movement direction detected for the plurality of receptive fields.

8. The apparatus of claim 7, wherein the predetermined receptive fields correspond to divided areas of the vision sensor, overlap an area of another neighboring receptive field, and are provided in a form of a circle.

9. The apparatus of claim 7, wherein the first direction detection filter comprises:

at least one winner takes all circuit which corresponds to each of the predetermined receptive filters, and

the winner takes all circuit outputs the movement direction of the object from a corresponding receptive filter, using a number of neurons which correspond to a number of directions determined by the second motion determiner.

10. The apparatus of claim 7, wherein the second direction detection filter comprises:

a winner takes all circuit configured to determine the final movement direction of the object based on the movement direction detected for the plurality of receptive fields, and

the winner takes all circuit is configured to output the final movement direction of the object based on the movement direction detected for the plurality of predetermined receptive fields using a number of neurons which correspond to the number of directions determined by the second motion determiner.

11. The apparatus of claim 7, wherein the second direction detection filter is configured to determine the final direction of movement of the object by calculating a vector sum of the movement direction detected for the plurality of predetermined receptive fields.

12. The apparatus of claim 1, further comprising:

a third motion determiner configured to classify the events outputted through the vision sensor into an ON event and an OFF event, calculate a rate of the OFF event, and determine progression or regression of an object by comparing the rate to at least one predetermined reference value in response to a result of the movement type determiner indicating a small movement.

13. An apparatus for recognizing motion using an event-based vision sensor, the apparatus comprising:

a vision sensor to sense a movement-occurring part and output events;

a first direction detection filter configured to receive an input of events outputted from a plurality of predetermined receptive fields of the vision sensor, and detect a movement direction in which an object moves, for the plurality of predetermined receptive fields;

a second direction detection filter configured to determine a final direction of movement of the object based on the direction of movement detected for the plurality of predetermined receptive fields; and

a motion controller configured to output a control instruction for controlling a device based on the determined final movement direction.

14. The apparatus of claim 13, wherein the predetermined receptive fields correspond to divided areas of the vision sensor, overlap an area of another neighboring receptive field, and are provided in a form of a circle.

15. The apparatus of claim 13, wherein the first direction detection filter comprises:

at least one winner takes all circuit which corresponds to each of the predetermined receptive fields, and

the winner takes all circuit is configured to output the movement direction of the object from a corresponding receptive filter, using a number of neurons which correspond to a number of directions determined by the second motion determiner.

16. The apparatus of claim 13, wherein the second direction detection filter comprises:

a winner takes all circuit to determine the final movement direction of the object based on the movement direction detected for the plurality of predetermined receptive fields, and

the winner takes all circuit outputs the final movement direction of the object based on the movement direction detected for the plurality of predetermined receptive fields using a number of neurons corresponding to the number of directions determined by the second direction detection filter of a second motion determiner.

17. A method of recognizing motion, the method comprising:

receiving an input of an event which corresponds to a movement-occurring part from a vision sensor;

determining a type of movement based on a frequency of occurrence of an event;

tracking a movement trajectory of the movement-occurring part and determining a motion pattern based on the movement trajectory in response to a result of determining the type of movement indicating a small movement;

determining a movement direction in which an object moves based on the events received from the vision sensor in response to a result of the determining the type of movement indicating a large movement; and

controlling a device based on the determined motion pattern or the determined direction of movement.

18. The method of claim 17, wherein the determining of the motion pattern comprises:

receiving an input of events outputted from a plurality of predetermined receptive fields of the vision sensor, and calculating a spatiotemporal correlation of spatiotemporal correlators which correspond to each of the predetermined receptive fields;

tracking a movement trajectory of the movement-occurring part based on a fluctuation in a spatiotemporal correlation for each of the predetermined receptive fields; and

determining the motion pattern of the movement-occurring part from the tracked movement trajectory.

19. The method of claim 18, wherein the tracking of the movement trajectory comprises generating a cluster by grouping spatiotemporal correlators having a high spatiotemporal correlation from among the spatiotemporal correlators, setting a predetermined area including the cluster to be a clustering area, calculating a center of the clustering area to be a location of an object, and tracking a movement trajectory in conjunction with the calculated location of the object.

20. The method of claim 19, wherein the tracking of the movement trajectory comprises re-calculating a center of a previous clustering area based on an internal state value of spatiotemporal correlators in the previous clustering area, calculating the center of the re-calculated previous clustering area to be a location of the object, and tracking a movement trajectory in conjunction with the calculated location of the object in response to the spatiotemporal correlators having the high spatiotemporal correlation being absent among the spatiotemporal correlators.

21. The method of claim 17, wherein the determining of the movement direction comprises:

detecting the movement direction in which the object moves for a plurality of predetermined receptive fields based on the events outputted from the predetermined receptive fields of the vision sensor; and

determining a final movement direction of the object based on the movement direction detected for the plurality of predetermined receptive fields.

22. A method of recognizing motion, the method comprising:

receiving an input of events outputted from a plurality of predetermined receptive fields of a vision sensor, wherein the predetermined receptive fields are areas of the vision sensor, the areas containing sensing units that output the events;

detecting a movement direction in which an object moves, for the plurality of predetermined receptive fields of the vision sensor;

determining a final movement direction of the object based on the movement direction detected for the plurality of predetermined receptive fields; and

controlling a device based on the determined final movement direction.

23. An apparatus for recognizing motion using an event-based vision sensor, the apparatus comprising:

a vision sensor configured to sense a movement-occurring part and output events; and

a plurality of direction detection filters provided in a hierarchical structure that are configured to receive an input of events outputted from predetermined receptive fields of the vision sensor and recognize a direction of movement,

wherein the plurality of direction detection filters comprises a first direction detection filter and a second direction detection filter, the first direction detection filter comprising a plurality of winner-take-all circuits configured to receive the input of events and to provide outputs, and the second direction detection filter comprising a final winner take all circuit configured to receive the outputs of the plurality of winner-take-all circuits as inputs.