Detecting and labeling places using runtime change-point detection

• US 8,565,538 B2
• Filed: 03/09/2011
• Issued: 10/22/2013
• Est. Priority Date: 03/16/2010
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ranganathan, Ananth
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): YEH, EUENG NAN
• Application Number:
  US13/044,287
• Publication Number:
  US 20110229032A1
• Time in Patent Office: 958 Days
• Field of Search: 382/171, 382/190, 382/225, 382/275, 348/335
• US Class Current: 382/225
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A system and method are disclosed for detecting and labeling places recognized in a video stream using change-points detection. The system includes a segmentation module and a label learning module. The segmentation module is configured to receive a video stream comprising multiple digital representations of images. The video stream is represented by a measurement stream comprising one or more image histograms of the video stream. The segmentation module segments the measurement stream into multiple segments corresponding to place recognized in the videos stream. The segmentation module detects change-points of the measurement stream and computes probability distributions of the segments over multiple pre-learned place models. The label generation module is configured to generate place labels for the places recognized by the place models.

Claims

1. A computer implemented method for labeling places recognized in a video stream, the method comprising:

receiving a video stream comprising multiple digital representations of images;

generating a measurement stream comprising one or more image histograms of the video stream, wherein generating the measurement stream comprises, for each image of the video stream;

extracting image features from the image;

generating one or more image histograms based on the extracted image features associated with the image, wherein each bin of an image histogram indicates the frequency of quantized image features in the image; and

generating a spatial pyramid of image histograms associated with the image, wherein the spatial pyramid of image histograms associated with the image is a measurement of the image in the measurement stream;

segmenting the measurement stream into a plurality of segments corresponding to types of places in the video stream based on the histograms of the videos stream, wherein the boundary between two adjacent segments represents a change-point of the measurement stream at a time-step;

computing probability distributions of the segments over a plurality of place models, wherein probability of a segment represents a likelihood that the segment is recognized by a place model of the plurality of the place models; and

generating place labels for the types of places recognized in video stream based on probabilities of change-points of the measurement stream, wherein the change-points are detected based on the probability distributions of the segments over the plurality of place models.

2. The method of claim 1, wherein generating one or more image histograms of the image of the videos stream comprises:

dividing the image into a plurality of sub-images, wherein each sub-image has a smaller spatial resolution than the spatial resolution of the image;

for each sub-image, generating one or more image histograms based on image features associated with the sub-image.

3. The method of claim 1, wherein segmenting the measurement stream comprises:

dividing the segment stream into a plurality of non-overlapping and adjacent segments;

modeling the change-points of the measurement stream by the plurality of the place models.

4. The method of claim 1, further comprising learning the plurality of place models from a plurality of image representations of images of an image training set.

5. The method of claim 4, wherein learning the plurality of place models comprises:

extracting image features of the images of the image training set, wherein the types of places recognized in each image of the image training set is pre-labeled;

clustering the extracted images features to create a codebook of a plurality of code words, wherein a code word is represented by a cluster identification of quantized image features associated with the code word;

generating one or more image histograms for each image of the image training set, wherein each bin of an image histogram indicates the frequency of quantized image features in the image.

6. The method of claim 1, wherein generating place labels for the types of places recognized in video stream comprises:

generating a place label for each segment of the measurement stream, wherein the place label has a probability indicting that the segment is recognized by one of the plurality of the place models;

comparing the probabilities for the segment recognized by the one or more place models;

selecting the place label that has the largest probability by one of the plurality of the place models.

7. The method of claim 1, wherein generating place labels for the types of places recognized in video stream further comprises:

responsive to a segment not recognized by the plurality of place models;

determining the segment corresponding to a place unknown to the plurality of the place models; and

assigning a new place label to the segment.

8. A computer system for labeling places recognized in a video stream, the system comprising:

a segmentation module configured to;

receive a video stream comprising multiple digital representations of images;

generate a measurement stream comprising one or more image histograms of the video stream, wherein generating the measurement stream comprises, for each image of the video stream;

extracting image features from the image;

generating one or more image histograms based on the extracted image features associated with the image, wherein each bin of an image histogram indicates the frequency of quantized image features in the image; and

generating a spatial pyramid of image histograms associated with the image, wherein the spatial pyramid of image histograms associated with the image is a measurement of the image in the measurement stream;

segment the measurement stream into a plurality of segments corresponding to types of places in the video stream based on the histograms of the videos stream, wherein the boundary between two adjacent segments represents a change-point of the measurement stream at a time-step;

compute probability distributions of the segments over a plurality of place models, wherein probability of a segment represents a likelihood that the segment is recognized by a place model of the plurality of the place models; and

a place label generation module configured to;

generate place labels for the types of places recognized in video stream based on probabilities of change-points of the measurement stream, wherein the change-points are detected based on the probability distributions of the segments over the plurality of place models.

9. The system of claim 8, wherein the segmentation module is further configured to:

divide the image into a plurality of sub-images, wherein each sub-image has a smaller spatial resolution than the spatial resolution of the image;

for each sub-image, generate one or more image histograms based on image features associated with the sub-image.

10. The system of claim 9, wherein the segmentation module is further configured to:

divide the segment stream into a plurality of non-overlapping and adjacent segments;

model the change-points of the measurement stream by the plurality of the place models.

11. The system of claim 8, further comprising a label learning module configured learn the plurality of place models from a plurality of image representations of images of an image training set.

12. The system of claim 11, wherein comprising:

a feature detection module configured to extract image features of the images of the image training set, wherein the types of places recognized in each image of the image training set is pre-labeled;

a feature clustering module configured to cluster the extracted images features to create a codebook of a plurality of code words, wherein a code word is represented by a cluster identification of the quantized image features associated with the code word;

an image representation module configured to generate one or more image histograms for each image of the image training set, wherein each bin of an image histogram indicates the frequency of quantized image features in the image.

13. The system of claim 8, wherein the label generating module is further configured to:

generate a place label for each segment of the measurement stream, wherein the place label has a probability indicting that the segment is recognized by one of the plurality of the place models;

compare the probabilities for the segment recognized by the one or more place models;

select the place label that has the largest probability by one of the plurality of the place models.

14. The system of claim 8, wherein the label generation module is further configured to:

responsive to a segment not recognized by the plurality of place models;

determine the segment corresponding to a place unknown to the plurality of the place models; and

assign a new place label to the segment.

15. A non-transitory computer program product for labeling places recognized in a video stream, the computer program product comprising a non-transitory computer-readable medium containing computer program code for performing the operations:

receiving a video stream comprising multiple digital representations of images;

generating a measurement stream comprising one or more image histograms of the video stream, wherein generating the measurement stream comprises, for each image of the video stream;

extracting image features from the image;

generating one or more image histograms based on the extracted image features associated with the image, wherein each bin of an image histogram indicates the frequency of quantized image features in the image; and

generating a spatial pyramid of image histograms associated with the image, wherein the spatial pyramid of image histograms associated with the image is a measurement of the image in the measurement stream;

segmenting the measurement stream into a plurality of segments corresponding to types of places in the video stream based on the histograms of the videos stream, wherein the boundary between two adjacent segments represents a change-point of the measurement stream at a time-step;

computing probability distributions of the segments over a plurality of place models, wherein probability of a segment represents a likelihood that the segment is recognized by a place model of the plurality of the place models; and

generating place labels for the types of places recognized in video stream based on probabilities of change-points of the measurement stream, wherein the change-points are detected based on the probability distributions of the segments over the plurality of place models.

16. The computer program product of claim 15, wherein the computer program code for generating one or more image histograms of the image of the videos stream comprises computer program code for:

dividing the image into a plurality of sub-images, wherein each sub-image has a smaller spatial resolution than the spatial resolution of the image;

for each sub-image, generating one or more image histograms based on image features associated with the sub-image.

17. The computer program product of claim 15, wherein the computer program code for segmenting the measurement stream comprises computer program code for:

dividing the segment stream into a plurality of non-overlapping and adjacent segments;

modeling the change-points of the measurement stream by the plurality of the place models.

18. The computer program product of claim 15, further comprising computer program code for learning the plurality of place models from a plurality of image representations of images of an image training set.

19. The computer program product of claim 18, wherein the computer program code for learning the plurality of place models comprises computer program code for:

extracting image features of the images of the image training set, wherein the types of places recognized in each image of the image training set is pre-labeled;

clustering the extracted images features to create a codebook of a plurality of code words, wherein a code word is represented by a cluster identification of quantized image features associated with the code word;

generating one or more image histograms for each image of the image training set, wherein each bin of an image histogram indicates the frequency of quantized image features in the image.

20. The computer program product of claim 15, wherein computer program code for generating place labels for the types of places recognized in video stream comprises computer program code for:

generating a place label for each segment of the measurement stream, wherein the place label has a probability indicting that the segment is recognized by one of the plurality of the place models;

comparing the probabilities for the segment recognized by the one or more place models;

selecting the place label that has the largest probability by one of the plurality of the place models.

21. The computer program product of claim 15, wherein the computer program code for generating place labels for the types of places recognized in video stream further comprises computer program code for:

responsive to a segment not recognized by the plurality of place models;

determining the segment corresponding to a place unknown to the plurality of the place models; and

assigning a new place label to the segment.