Systems and methods for feature tracking

• US 9,208,608 B2
• Filed: 02/25/2013
• Issued: 12/08/2015
• Est. Priority Date: 05/23/2012
• Status: Active Grant

• Current Assignee: Luxottica Retail North America Incorporated (EssilorLuxottica SA)
• Original Assignee: Glasses.com (EssilorLuxottica SA)
• Inventors: Turetzky, Darren, Engle, Ryan Travis
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US13/775,764
• Publication Number:
  US 20130315487A1
• Time in Patent Office: 1,016 Days
• Field of Search: 382/195
• US Class Current: 1/1
• CPC Class Codes: G02C 13/003 Measuring during assembly or fitting of spectacles
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/30 Polynomial surface description
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A computer-implemented method for processing, by a graphical processor unit (GPU), a plurality of images of a user. A plurality of features detected by the GPU in a first image of the plurality of images of the user is selected. Each selected feature includes one or more pixels. In a second image of the plurality of images of the user, a search is performed for the plurality of features selected in the first image. A variance is calculated, on the GPU, for each selected feature found in the second image. The variance indicates a degree to which a portion of the second image varies from a corresponding portion of the first image. The calculated variance is stored in a variance file.

Claims

1. A computer-implemented method for processing, by a graphical processor unit (GPU), a plurality of images of a user, the method comprising:

selecting, via a device with a central processing unit (CPU) and the GPU, a plurality of features detected by the GPU in a first image of the plurality of images of the user, wherein each feature comprises one or more pixels;

in a second image of the plurality of images of the user, searching for the plurality of features selected in the first image;

calculating, on the GPU, a variance for each selected feature found in the second image, wherein the variance indicates a degree a portion of the second image varies from a corresponding portion of the first image;

storing the calculated variance in a variance file; and

performing on the GPU a cross-correlation algorithm, wherein performing the cross-correlation algorithm on the GPU comprises;

performing, via the GPU, a fast Fourier transform (FFT) on first and second images from the plurality of images of the user.

2. The method of claim 1, further comprising:

selecting from among the selected features in the first image one or more patches based on the calculated variance of each selected feature, wherein each patch comprises a square area of pixels centered on one of the selected features of the first image.

3. The method of claim 2, further comprising:

upon selecting the one or more patches in the first image, removing the first image from memory.

4. The method of claim 2, wherein selecting the one or more patches further comprises:

selecting the one or more patches based on a predetermined threshold of calculated variance.

5. The method of claim 2, wherein selecting the one or more patches further comprises:

selecting the one or more patches based on a predetermined number of patches.

6. The method of claim 1, further comprising:

dividing each variance into first and second elements; and

storing the first element in a first file and the second element in a second file.

7. The method of claim 2, further comprising:

performing on the GPU the cross-correlation algorithm to determine how a first patch, selected among the one or more patches, is positioned in the first and second images of the plurality of images of the user.

8. The method of claim 7, wherein performing the cross-correlation algorithm on the GPU comprises:

determining a pose of the user in the first and second images;

performing the FFT on the first patch, wherein the FFT of the first patch is stored in a third file;

wherein performing the FFT on the first and second images includes placing the first image in a real element of a complex number and placing the second image in an imaginary element of the complex number;

multiplying element-wise the FFT of the first patch by the FFT of the first and second images;

calculating an inverse FFT of the multiplied FFTs, resulting in a first score for the first image and a second score for the second image; and

normalizing the result of the cross-correlation by dividing both first and second scores by the calculated variance stored in the variance file.

9. The method of claim 8, further comprising:

performing on the GPU a second cross-correlation algorithm on third and fourth images of the plurality of images of the user using the FFT of the first patch stored in the third file to determine how the first patch is positioned in the third and fourth images.

10. The method of claim 7, wherein performing on the GPU the cross-correlation algorithm further comprises

performing on the GPU the cross-correlation of each selected patch simultaneously.

11. The method of claim 1, further comprising:

determining a position of the selected feature of the first patch as a point in a virtual three-dimensional (3-D) space.

12. A computing device configured to process, by a graphical processor unit (GPU), a plurality of images of a user, comprising:

the GPU;

memory in electronic communication with the GPU;

instructions stored in the memory, the instructions being executable by the GPU to;

select, via a device with a central processing unit (CPU) and the GPU, a plurality of features detected by the GPU in a first image of the plurality of images of the user, wherein each feature comprises one or more pixels;

in a second image of the plurality of images of the user, search for the plurality of features selected in the first image;

calculate, on the GPU, a variance for each selected feature found in the second image, wherein the variance indicates a degree a portion of the second image varies from a corresponding portion of the first image;

storing the calculated variance in a variance file; and

perform on the GPU a cross-correlation algorithm, wherein performing the cross-correlation algorithm on the GPU comprises instructions executable by the GPU to;

perform, via the GPU, a fast Fourier transform (FFT) on first and second images from the plurality of images of the user, placing the first image in a real element of a complex number and placing the second image in an imaginary element of the complex number.

13. The computing device of claim 12, wherein the instructions are executable by the GPU to:

select from among the selected features in the first image one or more patches based on the calculated variance of each selected feature, wherein each patch comprises a square area of pixels centered on one of the selected features of the first image;

upon selecting the one or more patches in the first image, remove the first image from memory.

14. The computing device of claim 13, wherein the instructions are executable by the GPU to:

select the one or more patches based on a predetermined threshold of calculated variance.

15. The computing device of claim 13, wherein the instructions are executable by the GPU to:

perform on the GPU the cross-correlation algorithm to determine how a first patch, selected among the one or more patches, is positioned in the first and second images of the plurality of images of the user.

16. The computing device of claim 15, wherein performing the cross-correlation algorithm on the GPU comprises instructions executable by the GPU to:

determine a pose of the user in the first and second images;

perform the FFT on the first patch, wherein the FFT of the first patch is stored in a third file;

multiply element-wise the FFT of the first patch by the FFT of the first and second images;

calculate an inverse FFT of the multiplied FFTs, resulting in a first score for the first image and a second score for the second image; and

normalize the result of the cross-correlation by dividing both first and second scores by the calculated variance stored in the variance file.

17. The computing device of claim 16, wherein the instructions are executable by the GPU to:

perform on the GPU a second cross-correlation algorithm on third and fourth images of the plurality of images of the user using the FFT of the first patch stored in the third file to determine how the first patch is positioned in the third and fourth images.

18. The computing device of claim 15, wherein the instructions are executable by the GPU to:

perform on the GPU the cross-correlation of each selected patch simultaneously.

19. A computer-program product for processing, by a graphical processor unit (GPU), a plurality of images of a user, the computer-program product comprising a non-transitory computer-readable medium storing instructions thereon, the instructions being executable by the GPU to:

select, via a device with a central processing unit (CPU) and the GPU, a plurality of features detected by the GPU in a first image of the plurality of images of the user, wherein each feature comprises one or more pixels;

in a second image of the plurality of images of the user, search for the plurality of features selected in the first image;

calculate, on the GPU, a variance for each selected feature found in the second image, wherein the variance indicates a degree a portion of the second image varies from a corresponding portion of the first image;

store the calculated variance in a variance file; and

perform on the GPU a cross-correlation algorithm, wherein performing the cross-correlation algorithm on the GPU comprises instructions executable by the GPU to;

perform, via the GPU, a fast Fourier transform (FFT) on first and second images from the plurality of images of the user, placing the first image in a real element of a complex number and placing the second image in an imaginary element of the complex number.

20. The computer-program product of claim 19, wherein the instructions are executable by the GPU to:

perform on the GPU the first cross-correlation algorithm of each selected patch simultaneously to determine how each patch is positioned in the first and second images of the plurality of images of the user;

perform on the GPU a second cross-correlation algorithm of each selected patch simultaneously to determine how each patch is positioned in third and fourth images of the plurality of images of the user; and

normalize the results of the first and second cross-correlations by dividing the results by the calculated variance stored in the variance file.