Computationally Efficient Scene Classification

• US 20140355879A1
• Filed: 02/03/2014
• Published: 12/04/2014
• Est. Priority Date: 05/31/2013
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Agosta, John Mark, Oguchi, Kentaro, Pillai, Preeti, Yalla, Ganesh
• Granted Patent: US 9,129,161 B2
• Time in Patent Office: Days
• US Class Current: 382/170
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/422 for representing the structure of the pattern or shape of an object therefor
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

The disclosure describes novel technology for inferring scenes from images. In one example, the technology includes a system that can determine partition regions from one or more factors that are independent of the image data, for an image depicting a scene; receive image data including pixels forming the image; classify pixels of the image into one or more pixel types based on one or more pixel-level features; determine, for each partition region, a set of pixel characteristic data describing a portion of the image included in the partition region based on the one or more pixel types of pixels in the partition region; and classify a scene of the image based on the set of pixel characteristic data of each of the partition regions.

Claims

1. A computer-implemented method comprising:

determining partition regions for an image depicting a scene;

receiving image data including pixels forming the image;

classifying pixels of the image into one or more pixel types based on one or more pixel-level features;

determining, for each partition region, a set of pixel characteristic data describing a portion of the image included in the partition region based on the one or more pixel types of pixels in the partition region; and

classifying the scene of the image based on the set of pixel characteristic data of each of the partition regions.

2. The method of claim 1, wherein classifying the scene includes determining an overall impression of content depicted by the image and using the overall impression to categorize the image into one or more scene categories.

3. The method of claim 1, wherein at least two or more of the partition regions, when applied to the image, converge at a central point within an area of the image.

4. The method of claim 3, wherein the number of partition regions is five or more.

5. The method of claim 3, wherein the central point is a reference point that sets an expectation that determines one or more aspects used during classification of the scene, and a geometry of each of the partition regions is determined based on the central point.

6. The method of claim 1, wherein a geometry of the partition regions is determined based on one or more factors that are independent of the image data forming the image.

7. The method of claim 6, wherein the one or more factors include a motion of a moving platform, and the geometry of each of the partition regions is determined based at least on the motion of the moving platform.

8. The method of claim 3, wherein the central point is predictive of an epipole.

9. The method of claim 3, wherein each of the partition regions extends outwardly from the central point to an edge corresponding to an edge of the image.

10. The method of claim 1, further comprising:

capturing the image data from a moving platform traveling on a roadway, wherein the image depicts the scene including the roadway.

11. The method of claim 1, wherein determining the set of pixel characteristic data for each partition region comprises:

determining a pixel-type distribution within each partition region; and

generating the set of pixel characteristic data for each partition region based on the pixel-type distribution.

12. The method of claim 11, wherein the pixel-type distribution within each partition region is based on a histogram of the one or more pixel types within each partition region.

13. The method of claim 1, wherein classifying the scene of the image comprises:

determining a distribution associated with one or more scene variables based on the set of pixel characteristic data of each of the partition regions; and

determining a classification for the scene based on the distribution associated with the one or more scene variables.

14. A computer program product comprising a computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

determine partition regions for an image depicting a scene;

receive image data including pixels forming the image;

classify pixels of the image into one or more pixel types based on one or more pixel-level features;

determine, for each partition region, a set of pixel characteristic data describing a portion of the image included in the partition region based on the one or more pixel types of pixels in the partition region; and

classify a scene of the image based on the set of pixel characteristic data of each of the partition regions.

15. The computer program product of claim 14, wherein to classify the scene includes determining an overall impression of content depicted by the image and using the overall impression to categorize the image into one or more scene categories.

16. The computer program product of claim 14, wherein at least two or more of the partition regions, when applied to the image, converge at a central point within an area of the image.

17. The computer program product of claim 16, wherein the number of partition regions is five or more.

18. The computer program product of claim 16, wherein the central point is a reference point that sets an expectation that determines one or more aspects used during classification of the scene, and a geometry of each of the partition regions is determined based on the central point.

19. The computer program product of claim 16, wherein a geometry of the partition regions is determined based on one or more factors that are independent of the image data forming the image.

20. The computer program product of claim 19, wherein the one or more factors include a motion of a moving platform, and the geometry of each of the partition regions is determined based on the motion of the moving platform.

21. The computer program product of claim 16, wherein the central point is predictive of an epipole.

22. The computer program product of claim 16, wherein each of the partition regions extends outwardly from the central point to an edge corresponding to an edge of the image.

23. The computer program product of claim 14, wherein the computer readable program when executed on the computer causes the computer to:

capture the image data from a moving platform traveling on a roadway, wherein the image depicts the scene including the roadway.

24. The computer program product of claim 14, wherein determining the set of pixel characteristic data for each partition region comprises:

determining a pixel-type distribution within each partition region; and

generating the set of pixel characteristic data for each partition region based on the pixel-type distribution.

25. The computer program product of claim 24, wherein the pixel-type distribution within each partition region is based on a histogram of the one or more pixel types within each partition region.

26. The computer program product of claim 14, wherein to classify the scene of the image comprises:

determining a distribution associated with one or more scene variables based on the set of pixel characteristic data of each of the partition regions; and

determining a classification for the scene based on the distribution associated with the one or more scene variables.

27. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to;

determine partition regions for an image depicting a scene;

receive image data including pixels forming the image;

classify pixels of the image into one or more pixel types based on one or more pixel-level features;

determine, for each partition region, a set of pixel characteristic data describing a portion of the image included in the partition region based on the one or more pixel types of pixels in the partition region; and

classify a scene of the image based on the set of pixel characteristic data of each of the partition regions.

28. The system of claim 27, wherein the instructions cause the system to classify the scene by:

determining an overall impression of content depicted by the image and using the overall impression to categorize the image into one or more scene categories.

29. The system of claim 27, wherein at least two or more of the partition regions converge, when applied to the image, at a central point within an area of the image.

30. The system of claim 29, wherein the number of partition regions is five or more.

31. The system of claim 29, wherein the central point is a reference point that sets an expectation that determines one or more aspects used during classification of the scene, and a geometry of each of the partition regions is determined based on the central point.

32. The system of claim 29, wherein a geometry of the partition regions is determined based on one or more factors that are independent of the image data forming the image.

33. The system of claim 32, wherein the one or more factors include motion of a moving platform, and the geometry of each of the partition regions is determined based on the motion of the moving platform.

34. The system of claim 29, wherein the central point is predictive of an epipole.

35. The system of claim 29, wherein each of the partition regions extends outwardly from the central point to an edge corresponding to an edge of the image.

36. The system of claim 27, wherein the instructions when executed cause the system to:

capture the image data from a moving platform traveling on a roadway, wherein the image depicts the scene including the roadway.

37. The system of claim 27, wherein the instructions when executed cause the system to determine the set of pixel characteristic data for each partition region by:

determining a pixel-type distribution within each partition region; and

generating the set of pixel characteristic data for each partition region based on the pixel-type distribution.

38. The system of claim 37, wherein the pixel-type distribution within each partition region is based on a histogram of the one or more pixel types within each partition region.

39. The system of claim 27, wherein the instructions when executed cause the system to classify the scene of the image by:

determining a distribution associated with one or more scene variables based on the set of pixel characteristic data of each of the partition regions; and

determining a classification for the scene based on the distribution associated with the one or more scene variables.