METHOD AND APPARATUS FOR IMAGE SEGMENTATION USING AN EVENT SENSOR

• US 20200134827A1
• Filed: 10/28/2019
• Published: 04/30/2020
• Est. Priority Date: 10/26/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: SAHA, Sujoy, SRINIVASAN, Karthik, YADAV, Sourabh Singh, PALASAMUDRAM, Suhas Shantaraja, PEDDIGARI, Venkat Ramana, SK, Pradeep Kumar, DESHPANDE, Pranav P, KAR, Akankshya
• Granted Patent: US 11,263,751 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition

Abstract

A method for real-time semantic image segmentation using a monocular event-based sensor includes capturing a scene using a red, green, blue (RGB) sensor to obtain a plurality of RGB frames and an event sensor to obtain event data corresponding to each of the plurality of RGB frames, performing object labeling for objects in a first RGB frame among the plurality of RGB frames by identifying one or more object classes, obtaining an event velocity of the scene by fusing the event data corresponding to the first RGB frame and at least one subsequent RGB frame among the plurality of RGB frames, determining whether the event velocity is greater than a predefined threshold, and performing object labeling for objects in the at least one subsequent RGB frame based on the determination.

Claims

1. A method for image segmentation in an electronic device, the method comprising:

obtaining a first RGB frame of a scene using a sensor, the sensor comprising;

a red, green, blue (RGB) sensor generating RGB data, and

an event sensor generating event data corresponding to the first RGB frame;

classifying objects in the scene to one or more classes by identifying the one or more classes of objects in the scene;

obtaining event data corresponding to a second RGB frame with the event sensor;

identifying motion information of the scene with the classified objects based on the first RGB frame and the event data of the second RGB frame; and

updating an object boundary for the second RGB frame using the motion information.

2. The method of claim 1, further comprising:

predicting the object boundary of the second RGB frame using the one or more classes of objects in the scene.

3. The method of claim 1, wherein the event data of the scene is associated with motions of objects in the scene.

4. The method of claim 1, wherein the event data is updated with events that are detected by changes of pixel intensity in an image of the scene.

5. The method of claim 4, further comprising:

accumulating the events over a period of time; and

correlating the accumulated events with RGB frames including the first RGB frame and the second RGB frame.

6. The method of claim 5, further comprising:

determining an optimum sensitivity level of the event sensor among multiple sensitivity levels to identify spatially inconsistent regions based on the event data including the accumulated events.

7. The method of claim 5, wherein the identifying of the motion information comprises identifying the motion information of the scene with the accumulated events.

8. The method of claim 5, wherein the classifying of the objects in the scene to the one or more classes comprises:

generating an event velocity of the scene with changes in the accumulated events;

determining whether the event velocity is greater than a predefined threshold value; and

classifying the objects in the scene to the one or more classes in response to the determination.

9. The method of claim 1, further comprising:

identifying spatially inconsistent regions in the scene in response to obtaining the event data by changing sensitivity levels of the event sensor; and

masking the identified spatially inconsistent regions in an image of the scene.

10. The method of claim 1, wherein the obtaining of the event data of the second RGB frame comprises obtaining the event data corresponding to the second RGB frame at multiple sensitivity levels by changing exposure values of the event sensor.

11. The method of claim 1, wherein the identifying of the motion information of the scene comprises identifying the motion information of the scene by detecting changes in the object boundary with the event data of the second RGB frame.

12. The method of claim 1, wherein the event sensor captures the scene at a higher frame rate than the RGB sensor.

13. A method for image segmentation in an electronic device, the method comprising:

capturing a scene using a red, green, blue (RGB) sensor to obtain a plurality of RGB frames and an event sensor to obtain event data corresponding to each of the plurality of RGB frames,

performing object labeling for objects in a first RGB frame among the plurality of RGB frames by identifying one or more object classes;

obtaining an event velocity of the scene by fusing the event data corresponding to the first RGB frame and at least one subsequent RGB frame among the plurality of RGB frames;

determining whether the event velocity is greater than a predefined threshold; and

performing object labeling for objects in the at least one subsequent RGB frame based on the determination.

14. The method of claim 13, wherein the performing of the object labeling comprises:

classifying the objects into the one or more object classes.

15. The method of claim 13, further comprising:

identifying at least one object class in the captured scene during the performing of the object labeling for the objects in the first RGB frame;

predicting at least one object class in the at least one subsequent RGB frame using the at least one object labeling in the first RGB frame; and

updating the at least one object class of the at least one subsequent RGB frame using the event velocity of the scene.

16. The method of claim 15, further comprising:

extracting at least one feature from the scene using the event data accumulated with the first RGB frame and the at least one subsequent RGB frame;

correlating the at least one feature with the at least one subsequent RGB frame to identify spatially inconsistent regions in the scene; and

correcting the prediction of the at least one object class using the spatially inconsistent regions.

17. The method of claim 16, wherein the extracting of the at least one feature comprises extracting the at least one feature using the event data obtained at different sensitivity levels of the event sensor.

18. The method of claim 13, further comprising:

generating the event data at a plurality of sensitivity levels by changing exposure values of the event sensor.

19. An apparatus for image segmentation, the apparatus comprising:

a sensor configured to obtain a first red, green, blue (RGB) frame of a scene, the sensor comprising;

an RGB sensor configured to generate RGB data, and

an event sensor configured to generate event data of the first RGB frame and a second RGB frame; and

a processor configured to;

classify objects in the scene to one or more classes by identifying the one or more classes of objects in the scene,

identify motion information of the scene with the classified objects based on the first RGB frame and the event data of the second RGB frame; and

update an object boundary for the second RGB frame using the motion information.

20. The apparatus of claim 19, wherein the processor is configured to:

identify spatially inconsistent regions in the scene in response to obtaining the event data by changing sensitivity levels of the event sensor, and

control to mask the identified spatially inconsistent regions in an image of the scene.