Joint semantic segmentation of images and scan data

• US 8,494,285 B2
• Filed: 12/09/2011
• Issued: 07/23/2013
• Est. Priority Date: 12/09/2010
• Status: Active Grant

• Current Assignee: Hong Kong University of Science and Technology
• Original Assignee: Hong Kong University of Science and Technology
• Inventors: Zhang, Honghui, Quan, Long
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US13/315,587
• Publication Number:
  US 20120148162A1
• Time in Patent Office: 592 Days
• Field of Search: 382/154, 382/195, 382/284, 382/285
• US Class Current: 382/195
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods

Abstract

Systems, methods, and apparatus are described that that increase computer vision analysis in the field of semantic segmentation. With images accompanied by scan data, both two-dimensional and three-dimensional image information is employed for joint segmentation. Through the established correspondence between image data and scan data, two-dimensional and three-dimensional information respectively associated therewith is integrated. Using trained random forest classifiers, the probability of each pixel in images belonging to different object classes is predicted. With the predicted probability, optimization of the labeling of images and scan data is performed by integrating multiples cues in the markov random field.

Claims

1. A method comprising:

employing at least one processor executing computer executable instructions embodied on at least one computer readable medium to facilitate performing operations comprising;

receiving an image comprising image pixels comprising respective two-dimensional image features;

receiving image data points corresponding to the image, the image data points comprising respective three-dimensional image features;

correlating the image data points to corresponding ones of the image pixels associated with the image data points;

combining the respective two-dimensional image features and the respective three-dimensional image features of the image data points and the corresponding ones of the image pixels associated with the image data points and generating joint features for the image data points and the corresponding ones of the image pixels associated with the image data points; and

classifying the image pixels based on analyzing the joint features.

2. The method of claim 1, further comprising, classifying the image data points based on the analyzing.

3. The method of claim 1, wherein the correlating the image data points to the corresponding ones of the image pixels comprises:

grouping the image pixels into pixel groups comprising two or more of the image pixels; and

projecting the image data points to corresponding ones of the pixel groups associated with the image data points.

4. The method of claim 3, wherein the combining the respective two-dimensional image features and the respective three-dimensional image features of the image data points and the corresponding ones of the image pixels associated with the image data points comprises:

combining the respective two-dimensional image features of each image pixel in a pixel group with the respective three-dimensional image features of the image data points projected to the corresponding ones the of pixel groups associated with the image data points.

5. The method of claim 3, wherein the pixel groups comprise from about 100 to about 300 image pixels of the image pixels.

6. The method of claim 1, wherein the analyzing further comprises analyzing the respective two-dimensional image features of the image pixels not included in the corresponding ones of the image pixels associated with the image data points.

7. The method of claim 6, wherein the analyzing further comprises predicting probabilities that the image pixels belong to respective object classes.

8. The method of claim 1, wherein the analyzing further comprises predicting probabilities that the image pixels belong to respective object classes.

9. The method of claim 8, wherein the predicting the probabilities comprises employing trained random forest classifiers.

10. The method of claim 8, wherein the analyzing further comprises:

constructing a markov random field wherein two-dimensional and three-dimensional spatial smoothness constraints are encoded in a smooth term; and

employing the markov random field to determine a respective object class, from the respective object classes, to which the image pixels belong.

11. The method of claim 2, wherein the analyzing further comprises predicting probabilities that the corresponding ones of the image pixels associated with the image data points belong to respective object classes.

12. The method of claim 11, wherein the analyzing further comprises:

constructing a markov random field wherein three-dimensional spatial smoothness constraints are encoded in a smooth term; and

employing the markov random field to determine a respective object class, from the respective object classes, to which the image data points belong.

13. The method of claim 1, wherein the classifying image pixels comprises, classifying the image pixels as at least one of:

a building, a road, a vehicle, a tree, a person, or another structure, wherein another structure includes objects that are not a building, a road, a vehicle, a tree, and a person.

14. The method of claim 1, wherein the receiving the image data points comprises receiving laser scan data.

15. The method of claim 1 wherein the respective two dimensional image features include at least one of Texton features, dense scale-invariant feature transform (SIFT) features or two-dimensional positions of pixels in images.

16. The method of claim 1, wherein the respective three-dimensional image features include at least one of normal direction or height above ground.

17. The method of claim 16, further comprising:

estimating the respective three-dimensional image features.

18. A system, comprising:

a memory having computer executable components stored thereon; and

a processor communicatively coupled to the memory, the processor configured to facilitate execution of the computer executable components, the computer executable components, comprising;

an input component configured to receive an image comprising image pixels comprising respective two-dimensional image features, and to receive image data points corresponding to the image, the image data points comprising respective three-dimensional image features;

a correlation component configured to correlate the image data points to corresponding ones of the image pixels associated with the image data points;

an extraction component configured to combine the respective two-dimensional image features and the respective three-dimensional image features of the image data points and the corresponding ones of the image pixels associated with the image data points and form joint features for the image data points and the corresponding ones of the image pixels associated with the image data points;

an analysis component configured to analyze the joint features; and

a classification component configured to classify the image pixels based on the analysis.

19. The system of claim 18, wherein the classification component is further configured to classify the image data points based on the analysis of the joint features.

20. The system of claim 18, wherein the correlation component is further configured to group the image pixels into pixel groups comprising two or more of the image pixels, and project the image data points to corresponding ones the of the pixel groups associated with the image data points.

21. The system of claim 20, wherein the extraction component is further configured to combine the two-dimensional image features of each image pixel in a pixel group with the respective three-dimensional image features of the image data points projected to the corresponding ones the of pixel groups associated with the image data points.

22. The system of claim 20, wherein the pixel groups comprise from about 100 to about 300 of the image pixels.

23. The system of claim 18, wherein the analysis component is further configured to analyze the respective two-dimensional image features of the image pixels not included in the corresponding ones of the image pixels associated with the image data points, and wherein the classification component is further configured to classify the image pixels based on the analysis of the respective two-dimensional features.

24. The system of claim 23, wherein the analysis component is further configured to predict probabilities that the image pixels belong to respective object classes.

25. The system of claim 18, wherein the analysis component is further configured to predict probabilities that the image pixels belong to respective object classes.

26. The method of claim 25, wherein the analysis component is further configured to employ trained random forest classifiers to predict the probabilities.

27. The method of claim 25, wherein the analysis component is further configured to construct a markov random field wherein two-dimensional and three-dimensional spatial smoothness constraints are encoded in a smooth term, and employ the markov random field to determine a respective object class, from the respective object classes, to which the image pixels belong.

28. The system of claim 19, wherein the analysis component is further configured to predict probabilities that the corresponding ones of the image pixels associated with the image data points belong to respective object classes.

29. The system of claim 28, wherein the analysis component is further configured to construct a markov random field wherein three-dimensional spatial smoothness constraints are encoded in a smooth term and employ the markov random field to determine a respective object class, from the respective object classes, to which the image data points belong.

30. The system of claim 18, wherein the classification component is configured to classify the image pixels as at least one of:

a building, a road, a vehicle, a tree, a person, or another structure, wherein another structure includes objects that are not a building, a road, a vehicle, a tree, and a person.

31. The system of claim 18, wherein the input component is configured to receive the image data points as laser scan data.

32. The system of claim 18, wherein the respective two dimensional image features include at least one of Texton features, dense SIFT features or two-dimensional positions of pixels in images.

33. The system of claim 18, wherein the respective three-dimensional image features include at least one of normal direction or height above ground.

34. A computer-readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising:

receiving image data points corresponding to the image, the image data points comprising respective three-dimensional image features;

correlating the image data points to corresponding ones of the image pixels associated with the image data points;

combining the respective two-dimensional image features and the respective three-dimensional image features of the image data points and the corresponding ones of the image pixels associated with the image data points and generating joint features for the image data points and the corresponding ones of the image pixels associated with the image data points; and

classifying the image pixels based on analyzing the joint features.

35. The computer-readable storage medium claim 34, wherein the correlating the image data points to the corresponding ones of the image pixels comprises:

grouping the image pixels into pixel groups comprising two or more of the image pixels; and

projecting the image data points to corresponding ones the of the pixel groups associated with each of the image data points.

36. The computer-readable storage medium of claim 35, wherein the combining the respective two-dimensional image features and the respective three-dimensional image features of the image data points and the corresponding ones of the image pixels associated with the image data points comprises:

combining the respective two-dimensional image features of each image pixel in a pixel group with the respective three-dimensional image features of the image data points projected to the corresponding ones the of pixel groups associated with the image data points.

37. The computer-readable storage medium of claim 34, the operations further comprising:

analyzing the joint features; and

classifying at least one of the image pixels or the image data points based on the analyzing.

38. The computer-readable storage medium of claim 37, wherein the analyzing further comprises predicting probabilities that the image pixels belong to respective object classes.

39. The computer-readable storage medium of claim 38, wherein the analyzing further comprises:

constructing a markov random field wherein two-dimensional and three-dimensional spatial smoothness constraints are encoded in a smooth term; and

employing the markov random field to determine a respective object class, from the respective object classes, to which at least one of the image pixels or the image data points belong.