Missing feet recovery of a human object from an image sequence based on ground plane detection

• US 10,621,730 B2
• Filed: 05/22/2018
• Issued: 04/14/2020
• Est. Priority Date: 05/22/2018
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Takeda, Hiroyuki, Gharavi-Alkhansari, Mohammad
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/986,355
• Publication Number:
  US 20190362500A1
• Time in Patent Office: 693 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

An object segmentation system that includes a first type of sensor, a second type of sensor and a control circuitry. The first-type of sensor captures a sequence of color image frames of a scene. The second-type of sensor captures a depth image for each corresponding color image frame of the sequence of color image frames. The control circuitry generates a point cloud for an input color image frame. The control circuitry segments a foreground human object from a background of the input color image frame. The control circuitry detects a ground plane for the scene captured in the input color image frame. The control circuitry recovers a feet region in a defined region from a level of the detected ground plane. The control circuitry extracts the foreground human object with the recovered feet region from the background of the input color image frame, based on the detection of ground plane.

Claims

1. An object segmentation system, comprising:

a first-type of sensor configured to capture a sequence of color image frames of a scene;

a second-type of sensor configured to capture a depth image for each corresponding color image frame of the sequence of color image frames; and

control circuitry configured to;

generate a point cloud for an input color image frame of the sequence of color image frames received from the first-type of sensor, based on a corresponding input depth image for the input color image frame received from the second-type of sensor;

segment a foreground human object from a background of the input color image frame based on a first set of image parameters, wherein the segmented foreground human object is devoid of at least a portion of a feet region of the foreground human object;

detect a ground plane for the scene captured in the input color image frame by clusterization of a plurality of vectors projected from a plurality of points in the generated point cloud into a set of vector clusters, wherein the ground plane for the scene is detected based on orientation information associated with one vector cluster of the set of vector clusters in the generated point cloud;

recover the feet region in a defined region from a level of the detected ground plane in the input color image frame, based on a second set of image parameters that are different from the first set of image parameters; and

extract the foreground human object that includes the recovered feet region from the background of the input color image frame of the sequence of color image frames, based on the detection of the ground plane and a plurality of foreground pixels in the defined region from the level of the detected ground plane.

2. The system according to claim 1, wherein the control circuitry is further configured to:

detect a plurality of regions in the input depth image at which depth values are absent; and

smoothen the input depth image by a median filter and a uniform filter to fill the depth values at the plurality of regions and reduce noise in the input depth image.

3. The system according to claim 1, wherein the control circuitry is further configured to determine a three-dimensional (3D) coordinate of each point in the generated point cloud, based on the input depth image.

4. The system according to claim 3, wherein the control circuitry is further configured to compute the plurality of vectors projected from the plurality of points in the generated point cloud for the scene captured in the input color image frame, based on three-dimensional (3D) coordinates of a point-of-interest and two neighboring points of the point-of-interest in the plurality of points of the generated point cloud, wherein the computed plurality of vectors associated with the plurality of points in the generated point cloud are clustered into the set of vector clusters, based on orientation information associated with each of the computed plurality of vectors.

5. The system according to claim 4, wherein the control circuitry is further configured to detect a plurality of planes for the scene captured in the input color image frame, based on the set of vector clusters associated with the scene captured in the input color image frame.

6. The system according to claim 5, wherein the control circuitry is further configured to generate a plane map based on the set of vector clusters associated with the scene captured in the input color image frame, wherein the detected plurality of planes in image domain of the input color image frame is mapped to generate the plane map.

7. The system according to claim 6, wherein the plane map comprises pixels grouped in different planes of the detected plurality of planes in the plane map, based on same orientation information associated with each pixel in the plane map irrespective of spatial location of the pixels in the plane map in the image domain of the input color image frame of the scene.

8. The system according to claim 7, wherein the control circuitry is further configured to detect a plurality of isolated regions in the detected plurality of planes in the plane map, based on an area size of the plurality of isolated regions in the plane map that is less than a first threshold number of pixels in the image domain.

9. The system according to claim 8, wherein the control circuitry is further configured to eliminate the detected plurality of isolated regions in the plane map followed by a smoothening operation by a median filter to smoothen the detected plurality of isolated regions that are less than a second threshold number of pixels in the image domain.

10. The system according to claim 7, wherein the control circuitry is further configured to select a lowest plane in the detected plurality of planes in the plane map from a reference height for the scene, wherein the reference height corresponds to a height of the first-type of sensor that captures the sequence of color image frames of the scene, and wherein the selected lowest plane in the detected plurality of planes in the plane map is the ground plane for the scene captured in the input color image frame.

11. The system according to claim 7, wherein the control circuitry is further configured to:

segregate the pixels in different horizontal planes in the plane map based on a height-based distribution of the pixels in the plane map; and

classify pixels present in a lowest horizontal plane as compared to other pixels in the different horizontal planes as the pixels of the ground plane based on the height-based distribution of the pixels in the plane map.

12. The system according to claim 1, wherein the control circuitry is further configured to estimate a foreground mask of the foreground human object in the input color image frame of the scene using the input depth image and the first set of image parameters, and by binarization of a difference between the input color image frame and a background image of the input color image frame, wherein the foreground human object is segmented from the background of the input color image frame based on the estimated foreground mask.

13. The system according to claim 12, wherein the control circuitry is further configured to update the estimated foreground mask to include a set of pixels in the defined region from the level of the detected ground plane in the input color image frame, wherein the set of pixels corresponds to pixels of the feet region that are recovered based on the second set of image parameters.

14. A method, comprising:

in an object segmentation system comprising a first-type of sensor, a second-type of sensor, and a control circuitry;

capturing, by the first-type of sensor, a sequence of color image frames of a scene;

capturing, by the second-type of sensor, a depth image for each corresponding color image frame of the sequence of color image frames;

generating, by the control circuitry, a point cloud for an input color image frame of the sequence of color image frames received from the first-type of sensor, based on a corresponding input depth image for the input color image frame received from the second-type of sensor;

segmenting, by the control circuitry, a foreground human object from a background of the input color image frame based on a first set of image parameters, wherein the segmented foreground human object is devoid of at least a portion of a feet region of the foreground human object;

detecting, by the control circuitry, a ground plane for the scene captured in the input color image frame, by clusterization of a plurality of vectors projected from a plurality of points in the generated point cloud into a set of vector clusters, wherein the ground plane for the scene is detected based on orientation information associated with one vector cluster of the set of vector clusters in the generated point cloud;

recovering, by the control circuitry, the feet region in a defined region from a level of the detected ground plane in the input color image frame, based on a second set of image parameters that are different from the first set of image parameters; and

extracting, by the control circuitry, the foreground human object that includes the recovered feet region from the background of the input color image frame of the sequence of color image frames, based on the detection of the ground plane and a plurality of foreground pixels in the defined region from the level of the detected ground plane.

15. The method according to claim 14, further comprising:

detecting, by the control circuitry, a plurality of regions in the input depth image at which depth values are absent; and

smoothing, by the control circuitry, the input depth image by a median filter and a uniform filter to fill the depth values at the plurality of regions and reduce noise in the input depth image.

16. The method according to claim 14, further comprising determining, by the control circuitry, a three-dimensional (3D) coordinate of each point in the generated point cloud, based on the input depth image.

17. The method according to claim 16, further comprising computing, by the control circuitry, the plurality of vectors projected from the plurality of points in the generated point cloud for the scene captured in the input color image frame, based on three-dimensional (3D) coordinates of a point-of-interest and two neighboring points of the point-of-interest in the plurality of points of the generated point cloud.

18. The method according to claim 17, further comprising detecting, by the control circuitry, a plurality of planes for the scene captured in the input color image frame, based on the set of vector clusters associated with the scene captured in the input color image frame.

19. The method according to claim 18, further comprising generating, by the control circuitry, a plane map based on the set of vector clusters associated with the scene captured in the input color image frame, wherein the plane map is generated by mapping the detected plurality of planes in image domain of the input color image frame.

20. The method according to claim 19, wherein the plane map comprises pixels grouped in different planes of the detected plurality of planes in the plane map, based on same orientation information associated with each pixel in the plane map irrespective of spatial location of the pixels in the plane map in the image domain of the input color image frame of the scene.