Methods for assisting with object recognition in image sequences and devices thereof

• US 9,785,835 B2
• Filed: 03/22/2012
• Issued: 10/10/2017
• Est. Priority Date: 03/22/2011
• Status: Active Grant

• Current Assignee: Rochester Institute of Technology
• Original Assignee: Rochester Institute of Technology
• Inventors: Pelz, Jeff B., Kinsman, Thomas B., Pontillo, Daniel F., Munn, Susan M., Harrington, Nicholas R., Hsieh, Brendon Ben-Kan
• Primary Examiner(s): Conner, Sean
• Application Number:
  US13/427,519
• Publication Number:
  US 20120328150A1
• Time in Patent Office: 2,028 Days
• Field of Search: None
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A method, non-transitory computer readable medium, and apparatus that assist with object recognition includes determining when at least one eye of an observer fixates on a location in one or more of a sequence of fixation tracking images. The determined fixation location in the one or more of the sequence of images is correlated to a corresponding one of one or more sequence of field of view images. At least the determined fixation location in each of the correlated sequence of field of view images is classified based on at least one of a classification input or a measurement and comparison of one or more features of the determined fixation location in each of the correlated sequence of field of view images against one or more stored measurement feature values. The determined classification of the fixation location in each of the correlated sequence of field of view images is output.

Claims

1. An adaptive method for aiding a user in classifying objects from field of view images using eye fixation, the method comprising:

obtaining a sequence of fixation tracking images from an eye tracking video of one or both of an observer'"'"'s eyes and a sequence of field of view images from a view tracking video of what is in view of the observer, wherein the sequence of fixation tracking images are synchronized in time with the sequence of field of view images;

determining by an object recognition processing apparatus when in the sequence of fixation tracking images an eye of the observer fixates on a location in one or more images of the sequence of field of view images;

correlating by the object recognition processing apparatus the determined eye fixation to a location in the one or more images of the sequence of field of view images;

extracting image data surrounding the fixated location from the one or more images of the sequence of field of view images;

presenting an operator of the object recognition processing apparatus with the extracted image data;

specifying a label for an object within the presented image data to classify the object by i) the operator assigning a label to the object based upon the presented image data, ii) auto-accepting a label from labelled image data in a classification database for the object representing the closest match to the presented image data, or iii) showing the operator a ranked set of a plurality of labels from labelled image data in the classification database associated with the closest matches to the presented image data, wherein the operator selects one of the plurality of labels as the label for the object;

adding to the classification database a record of the presented image data comprising the specified label; and

repeating the above determining, correlating, extracting, presenting, specifying and adding steps on a further unidentified object from the extracted image data,

so as to build the classification database from the classified object and the further classified objects.

2. The method as set forth in claim 1 further comprising prescreening by the object recognition processing apparatus to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

3. The method as set forth in claim 1 wherein the classifying further comprises applying by the object recognition processing apparatus one or more weighting factors to a comparison of one or more measured features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values.

4. The method as set forth in claim 1 wherein the classifying further comprises:

determining by the object recognition processing apparatus one or more preliminary classifications of the determined fixated object in each of the correlated sequence of field of view images based on a measurement and comparison of the one or more features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

outputting by the object recognition processing apparatus the determined one or more preliminary classifications of the determined fixated object in the at least one of the correlated sequence of field of view images; and

receiving at the object recognition processing apparatus a selection of one of the determined one or more preliminary classifications as the classification of the determined fixated object in the at least one of the correlated sequence of field of view images.

5. The method as set forth in claim 1 wherein the classifying with the object recognition processing apparatus further comprises:

displaying by the object recognition processing apparatus the determined fixated object in at least one of the correlated sequence of field of view images; and

receiving at the object recognition processing apparatus a classification input of the determined fixated object in the at least one of the correlated sequence of field of view images based on the displaying.

6. The method as set forth in claim 1 wherein the classifying further comprises:

determining by the object recognition processing apparatus one or more preliminary classifications of the determined fixated object for at least one of the correlated sequence of field of view images based on a measurement and comparison of one or more of the features of the determined fixated object in the at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

determining by the object recognition processing apparatus an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixated object in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining by the object recognition processing apparatus when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining by the object recognition processing apparatus when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting by the object recognition processing apparatus the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixated object in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

7. The method as set forth in claim 1 wherein the determined fixated object in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

8. The method as set forth in claim 1 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

9. The method as set forth in claim 8 wherein the one or more features further comprise one or more location features.

10. A non-transitory computer readable medium having stored thereon instructions for aiding a user in classifying objects from field of view images using eye fixation comprising machine executable code which when executed by at least one processor, causes the processor to perform steps comprising:

obtaining a sequence of fixation tracking images from an eye tracking video of one or both of an observer'"'"'s eyes and a sequence of field of view images from a view tracking video of what is in view of the observer, wherein the sequence of fixation tracking images are synchronized in time with the sequence of field of view images;

determining when in the sequence of fixation tracking images at least one eye of an observer fixates on a location in one or more of a sequence of field of view images;

correlating the fixated object within the determined eye fixation to a location in the one or more images of the sequence of field of view images to a corresponding one of one or more sequence of field of view images;

extracting image data surrounding the fixated location from the one or more images of the sequence of field of view images;

presenting an operator of the object recognition processing apparatus with the extracted image data;

specifying a label for an object within the presented image data to classify the object by i) the operator assigning a label to the object based upon the presented image data, ii) auto-accepting a label from labelled image data in a classification database for the object representing the closest match to the presented image data, or iii) showing the operator a ranked set of a plurality of labels from labelled image data in the classification database associated with the closest matches to the presented image data, wherein the operator selects one of the plurality of labels as the label for the object;

adding to the classification database a record of the presented image data comprising the specified label; and

repeating the above determining, correlating, extracting, presenting, specifying and adding steps on a further unidentified object from the extracted image data,

so as to build the classification database from the classified object and the further classified objects.

11. The medium as set forth in claim 10 further comprising prescreening to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

12. The medium as set forth in claim 10 wherein the classifying further comprises applying one or more weighting factors to a comparison of one or more measured features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values.

13. The medium as set forth in claim 10 wherein the classifying further comprises:

determining one or more preliminary classifications of the determined fixated object in each of the correlated sequence of field of view images based on a measurement and comparison of the one or more features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

outputting the determined one or more preliminary classifications of the determined fixated object in the at least one of the correlated sequence of field of view images; and

receiving a selection of one of the determined one or more preliminary classifications as the classification of the determined fixated object in the at least one of the correlated sequence of field of view images.

14. The medium as set forth in claim 10 wherein the classifying further comprises:

displaying the determined fixated object in at least one of the correlated sequence of field of view images; and

receiving a classification input of the determined fixated object in the at least one of the correlated sequence of field of view images based on the displaying.

15. The medium as set forth in claim 10 wherein the classifying further comprises:

determining one or more preliminary classifications of the determined fixated object for at least one of the correlated sequence of field of view images based on a measurement and comparison of one or more of the features of the determined fixated object in the at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

determining an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixated object in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixated object in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

16. The medium as set forth in claim 10 wherein the determined fixated object in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

17. The medium as set forth in claim 10 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

18. The medium as set forth in claim 17 wherein the one or more features further comprise one or more location features.

19. An object recognition processing apparatus comprising:

one or more processors;

a memory coupled to the one or more processors which are configured to execute programmed instructions stored in the memory comprising;

obtaining a sequence of fixation tracking images from an eye tracking video of one or both of an observer'"'"'s eyes and a sequence of field of view images from a view tracking video of what is in view of the observer, wherein the sequence of fixation tracking images are synchronized in time with the sequence of field of view images;

determining when in the sequence of fixation tracking images at least one eye of an observer fixates on a location in one or more images of a sequence of field of view images;

correlating the fixated object within the determined eye fixation to a location in the one or more images of the sequence of field of view images to a corresponding one of one or more sequence of field of view images;

extracting image data surrounding the fixated location from the one or more images of the sequence of field of view images;

presenting an operator of the object recognition processing apparatus with the extracted image data;

specifying a label for an object within the presented image data to classify the object by i) the operator assigning a label to the object based upon the presented image data, ii) auto-accepting a label from labelled image data in a classification database for the object representing the closest match to the presented image data, or iii) showing the operator a ranked set of a plurality of labels from labelled image data in the classification database associated with the closest matches to the presented image data, wherein the operator selects one of the plurality of labels as the label for the object;

adding to the classification database a record of the presented image data comprising the specified label; and

repeating the above determining, correlating, extracting, presenting, specifying and adding steps on a further unidentified object from the extracted image data,

so as to build the classification database from the classified object and the further classified objects.

20. The apparatus as set forth in claim 19 wherein the processor is further configured to execute programmed instructions stored in the memory further comprising prescreening to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

21. The apparatus as set forth in claim 19 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising applying one or more weighting factors to a comparison of one or more measured features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values.

22. The apparatus as set forth in claim 19 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising:

determining one or more preliminary classifications of the determined fixated object in each of the correlated sequence of field of view images based on a measurement and comparison of the one or more features of the determined fixated object in at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

outputting the determined one or more preliminary classifications of the determined fixated object in the at least one of the correlated sequence of field of view images; and

receiving a selection of one of the determined one or more preliminary classifications as the classification of the determined fixated object in the at least one of the correlated sequence of field of view images.

23. The apparatus as set forth in claim 19 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprises:

displaying the determined fixated object in at least one of the correlated sequence of field of view images; and

receiving a classification input of the determined fixated object in the at least one of the correlated sequence of field of view images based on the displaying.

24. The apparatus as set forth in claim 19 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising:

determining one or more preliminary classifications of the determined fixated object for at least one of the correlated sequence of field of view images based on a measurement and comparison of one or more of the features of the determined fixated object in the at least one of the correlated sequence of field of view images against one or more stored measurement feature values;

determining an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixated object in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixated object in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

25. The apparatus as set forth in claim 19 wherein the determined fixated object in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

26. The apparatus as set forth in claim 19 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

27. The apparatus as set forth in claim 26 wherein the one or more features further comprise one or more location features.