METHODS FOR ASSISTING WITH OBJECT RECOGNITION IN IMAGE SEQUENCES AND DEVICES THEREOF

• US 20120328150A1
• Filed: 03/22/2012
• Published: 12/27/2012
• Est. Priority Date: 03/22/2011
• Status: Active Grant

• Current Assignee: Rochester Institute of Technology
• Original Assignee: Rochester Institute of Technology
• Inventors: Pelz, Jeff B., Kinsman, Thomas B., Pontillo, Daniel F., Munn, Susan M., Harrington, Nicholas R., Hsieh, Brendon Ben-Kan
• Granted Patent: US 9,785,835 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A method, non-transitory computer readable medium, and apparatus that assist with object recognition includes determining when at least one eye of an observer fixates on a location in one or more of a sequence of fixation tracking images. The determined fixation location in the one or more of the sequence of fixation tracking images is correlated to a corresponding one of one or more sequence of field of view images. At least the determined fixation location in each of the correlated sequence of field of view images is classified based on at least one of a classification input or a measurement and comparison of one or more features of the determined fixation location in each of the correlated sequence of field of view images against one or more stored measurement feature values. The determined classification of the fixation location in each of the correlated sequence of field of view images is output.

Claims

1. A method for assisting with object recognition, the method comprising:

determining by an object recognition processing apparatus when an observer fixates on a location in one or more of a sequence of fixation tracking images;

correlating by the object recognition processing apparatus the determined fixation location in the one or more of the sequence of fixation tracking images to a corresponding one of one or more sequence of field of view images;

classifying by the object recognition processing apparatus at least the determined fixation location in each of the correlated sequence of field of view images based on at least one of a classification input or a measurement and comparison of one or more features of the determined fixation location in each of the correlated sequence of field of view images against one or more stored measurement feature values; and

outputting with the object recognition processing apparatus the determined classification of the fixation location in each of the correlated sequence of field of view images.

2. The method as set forth in claim 1 further comprising obtaining by the object recognition processing apparatus the sequence of fixation tracking images and the sequence of field of view images in an overlapping time frame.

3. The method as set forth in claim 1 further comprising prescreening by the object recognition processing apparatus to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

4. The method as set forth in claim 1 wherein the classifying further comprises applying by the object recognition processing apparatus one or more weighting factors to the comparison of the one or more measured features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values.

5. The method as set forth in claim 1 wherein the classifying further comprises:

determining by the object recognition processing apparatus one or more preliminary classifications of the determined fixation location in each of the correlated sequence of field of view images based on the measurement and comparison of the one or more features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

outputting by the object recognition processing apparatus the determined one or more preliminary classifications of the determined fixation location in the at least one of the correlated sequence of field of view images; and

receiving at the object recognition processing apparatus a selection of one of the determined one or more preliminary classifications as the classification of the determined fixation location in the at least one of the correlated sequence of field of view images.

6. The method as set forth in claim 1 wherein the classifying with the object recognition processing apparatus based on the classification input further comprises:

displaying by the object recognition processing apparatus the determined fixation location in at least one of the correlated sequence of field of view images; and

receiving at the object recognition processing apparatus the classification input of the determined fixation location in the at least one of the correlated sequence of field of view images based on the displaying.

7. The method as set forth in claim 1 wherein the classifying further comprises:

determining by the object recognition processing apparatus one or more preliminary classifications of the determined fixation location for at least one of the correlated sequence of field of view images based on the measurement and comparison of one or more of the features of the determined fixation location in the at least one of the correlated sequence of field of view images against one or more of the stored measurement feature values;

determining by the object recognition processing apparatus an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixation location in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining by the object recognition processing apparatus when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining by the object recognition processing apparatus when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting by the object recognition processing apparatus the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixation location in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

8. The method as set forth in claim 1 wherein the determined fixation location in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

9. The method as set forth in claim 1 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

10. The method as set forth in claim 9 wherein the one or more features further comprise one or more location features.

11. A non-transitory computer readable medium having stored thereon instructions for assisting with object recognition comprising machine executable code which when executed by at least one processor, causes the processor to perform steps comprising:

determining when at least one eye of an observer fixates on a location in one or more of a sequence of fixation tracking images;

correlating the determined fixation location in the one or more of the sequence of fixation tracking images to a corresponding one of one or more sequence of field of view images;

classifying at least the determined fixation location in each of the correlated sequence of field of view images based on at least one of a classification input or a measurement and comparison of one or more features of the determined fixation location in each of the correlated sequence of field of view images against one or more stored measurement feature values; and

outputting the determined classification of the fixation location in each of the correlated sequence of field of view images.

12. The medium as set forth in claim 11 further comprising obtaining the sequence of fixation tracking images and the sequence of field of view images in an overlapping time frame.

13. The medium as set forth in claim 11 further comprising prescreening to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

14. The medium as set forth in claim 11 wherein the classifying further comprises applying one or more weighting factors to the comparison of the one or more measured features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values.

15. The medium as set forth in claim 11 wherein the classifying further comprises:

determining one or more preliminary classifications of the determined fixation location in each of the correlated sequence of field of view images based on the measurement and comparison of the one or more features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

outputting the determined one or more preliminary classifications of the determined fixation location in the at least one of the correlated sequence of field of view images; and

receiving a selection of one of the determined one or more preliminary classifications as the classification of the determined fixation location in the at least one of the correlated sequence of field of view images.

16. The medium as set forth in claim 11 wherein the classifying based on the classification input further comprises:

displaying the determined fixation location in at least one of the correlated sequence of field of view images; and

receiving the classification input of the determined fixation location in the at least one of the correlated sequence of field of view images based on the displaying.

17. The medium as set forth in claim 11 wherein the classifying further comprises:

determining one or more preliminary classifications of the determined fixation location for at least one of the correlated sequence of field of view images based on the measurement and comparison of one or more of the features of the determined fixation location in the at least one of the correlated sequence of field of view images against one or more of the stored measurement feature values;

determining an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixation location in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixation location in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

18. The medium as set forth in claim 11 wherein the determined fixation location in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

19. The medium as set forth in claim 11 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

20. The medium as set forth in claim 19 wherein the one or more features further comprise one or more location features.

21. An object recognition processing apparatus comprising:

one or more processors;

a memory coupled to the one or more processors which are configured to execute programmed instructions stored in the memory comprising;

determining when at least one eye of an observer fixates on a location in one or more of a sequence of fixation tracking images;

correlating the determined fixation location in the one or more of the sequence of fixation tracking images to a corresponding one of one or more sequence of field of view images;

classifying at least the determined fixation location in each of the correlated sequence of field of view images based on at least one of a classification input or a measurement and comparison of one or more features of the determined fixation location in each of the correlated sequence of field of view images against one or more stored measurement feature values; and

outputting the determined classification of the fixation location in each of the correlated sequence of field of view images.

22. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory further comprising obtaining the sequence of fixation tracking images and the sequence of field of view images in an overlapping time frame.

23. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory further comprising prescreening to eliminate one or more of the sequence of field of view images based on one or more criteria before the classifying.

24. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising applying one or more weighting factors to the comparison of the one or more measured features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values.

25. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising:

determining one or more preliminary classifications of the determined fixation location in each of the correlated sequence of field of view images based on the measurement and comparison of the one or more features of the determined fixation location in at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

outputting the determined one or more preliminary classifications of the determined fixation location in the at least one of the correlated sequence of field of view images; and

receiving a selection of one of the determined one or more preliminary classifications as the classification of the determined fixation location in the at least one of the correlated sequence of field of view images.

26. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying based on the classification input further comprises:

displaying the determined fixation location in at least one of the correlated sequence of field of view images; and

receiving the classification input of the determined fixation location in the at least one of the correlated sequence of field of view images based on the displaying.

27. The apparatus as set forth in claim 21 wherein the processor is further configured to execute programmed instructions stored in the memory for the classifying further comprising:

determining one or more preliminary classifications of the determined fixation location for at least one of the correlated sequence of field of view images based on the measurement and comparison of one or more of the features of the determined fixation location in the at least one of the correlated sequence of field of view images against one or more of the stored measurement feature values;

determining an accuracy rating for each of the determined one or more preliminary classifications based on each of the comparisons of the measured one or more features of the determined fixation location in the at least one of the correlated sequence of field of view images against the one or more stored measurement feature values;

determining when the determined accuracy rating with a highest value for one of the one or more preliminary classifications is above a first set threshold; and

determining when the determined accuracy rating with a next highest value for another one of the one or more preliminary classifications is below a second set threshold; and

setting the one of the one or more preliminary classifications with the determined accuracy rating with the highest value as the determined classification of the determined fixation location in the at least one of the correlated sequence of field of view images when the determined accuracy rating with the highest value is above the first set threshold and the determined accuracy rating with the next highest value is below the second set threshold.

28. The apparatus as set forth in claim 21 wherein the determined fixation location in at least one of the correlated one or more sequence of field of view images comprises a region in each of the correlated one or more sequence of field of view images.

29. The apparatus as set forth in claim 21 wherein the one or more features further comprise at least one of one or more spectral features, spatial features or temporal features.

30. The apparatus as set forth in claim 29 wherein the one or more features further comprise one or more location features.