Locating and Augmenting Object Features in Images

• US 20160196665A1
• Filed: 07/25/2014
• Published: 07/07/2016
• Est. Priority Date: 07/30/2013
• Status: Active Grant

• Current Assignee: Holition Limited
• Original Assignee: Holition Limited
• Inventors: Abreu, Joe, Garcia Sopo, Maria Jose
• Granted Patent: US 9,990,734 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A computer-implemented method and system are described for augmenting image data of an object in an image, the method comprising receiving captured image data from a camera, storing a plurality of augmentation image data defining a respective plurality of augmentation values to be applied to the captured image data, storing a plurality of augmentation representations, each representation identifying a respective portion of augmentation image data, selecting one of said augmentation image data and one of said augmentation representations based on at least one colourisation parameter, determining a portion of the augmentation image data to be applied based on the selected augmentation representation, augmenting the captured image data by applying said determined portion of the augmentation image data to the corresponding portion of the captured image data, and outputting the augmented captured image data.

Claims

1. A computer-implemented method of augmenting image data, the method comprising:

receiving data of an image captured by a camera;

receiving data identifying coordinates of a plurality of labelled feature points defining a detected object in the captured image;

storing at least one augmentation image data defining a plurality of augmentation values to be applied to the captured image data;

storing at least one augmentation representation including data defining at least one polygonal region of the augmentation image data, the or each polygonal region defined by three or more of vertices, each vertex associated with a corresponding labelled feature point;

determining a transformation of the at least one polygonal region of the representation based on the received coordinates of the corresponding feature points;

applying the determined transformation to corresponding regions of the augmentation image data defined by the at least one polygonal regions of the augmentation representation; and

augmenting the captured image data by applying the transformed at least one portion of the augmentation image data to the corresponding portion of the captured image data.

2. The method of claim 1, wherein the augmentation image data comprises one or more of texture image data, data identifying one or more material properties, and a mathematical model to generate an array of augmentation values.

3. The method of claim 1, wherein the augmentation image data and the captured image data have the same dimensions.

4. The method of claim 1, wherein the augmentation image data and the captured image data have different dimensions.

5. The method of claim 1, wherein coordinates of the vertices of the augmentation representation are defined relative to pixel locations of the augmentation image data.

6. The method of claim 1, wherein augmenting the captured image data further comprises applying at least one image data adjustment to the transformed at least one portion of the augmentation image data.

7. The method of claim 6, wherein the at least one image data adjustment comprises one or more of a highlight adjustment, a colour adjustment, a glitter adjustment, a lighting model adjustment, a blend colour adjustment, and an alpha blend adjustment.

8. The method of claim 1, further comprising storing mask data defining at least one masked portion of the augmentation image data.

9. The method of claim 8, further comprising applying the determined transformation to corresponding regions of the mask data defined by the at least one polygonal regions of the augmentation representation.

10. The method of claim 9, wherein the at least one image data adjustment comprises alpha blending said transformed masked data regions.

11. The method of claim 1, further comprising selecting a stored augmentation image data and a stored augmentation representation based on at least one colourisation parameter.

12. The method of claim 11, wherein a plurality of stored augmentation image data and stored augmentation representations are selected based on respective at least one colourisation parameters.

13. The method of claim 12, wherein augmenting the captured image data comprises alpha blending the results of applying, for each selected augmentation representation in sequence, transformed at least one portions of each selected augmentation image data to the corresponding portion of the captured image data, and applying the alpha blended output to the captured image data.

14. A computer-implemented method of generating the at least one augmentation representations as set out in claim 1, based on respective predefined mask data identifying coordinates of a plurality of masked pixels.

15. The method of claim 14, wherein generating an augmentation representation comprises:

retrieving data defining a plurality of polygonal regions determined for the detected object;

placing the retrieved plurality of polygonal regions over the respective mask data;

identifying polygonal regions that include at least one masked pixel; and

storing data representing the identified subset of polygonal regions.

16. The method of claim 14, wherein each masked pixel comprises a value representing a blend parameter.

17. The method of claim 1, further comprising locating the detected object in the captured image by:

storing a representation of the object, the representation including data defining a first object model and a corresponding function that approximates variations to the first object model, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding function that approximates variations to the respective second object model;

determining an approximate location of the object in the captured image, based on the first object model and its corresponding function; and

refining the location of the object in the captured image by determining a location of a portion of the object, based on the at least one second object model and its corresponding function.

18. The method of claim 17, wherein the first object model comprises data representing locations of a plurality of feature points and the second object model comprises a subset of the feature points of the first object model.

19. The method of claim 17, wherein the first object model defines a shape of the whole object and the at least one second object model defines a shape of a portion of the object.

20. The method of claim 17, wherein determining the approximate location of the object in the image comprises generating a candidate object shape based on the first object model and applying the corresponding function to determine an approximate location of the candidate object shape.

23. The method of claim 17, wherein the corresponding functions comprise regression coefficient matrices.

24. The method of claim 23, wherein the corresponding function that approximates variations to the second object model comprises a plurality of cascading regression coefficient matrices.

25. The method of claim 24, further comprising iteratively refining the location of the one or more candidate object sub-shapes based on the respective second object model and its corresponding plurality of cascading regression coefficient matrices.

21. The method of claim 21, further comprising splitting the candidate object into one or more candidate object sub-shapes based on the at least one second object models.

22. The method of claim 21, further comprising refining the location of the one or more candidate object sub-shapes based on the respective second object model and its corresponding function.

26. A computer-implemented method of locating an object in an image, the method comprising:

storing a representation of the object, the representation including data defining a first object model and a corresponding function that approximates variations to the first object model, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding function that approximates variations to the respective second object model;

determining an approximate location of the object in the image, based on the first object model and its corresponding function; and

refining the location of the object in the image by determining a location of a portion of the object, based on the at least one second object model and its corresponding function.

27. A system comprising one or more processors configured to perform the method of:

storing a representation of the object, the representation including data defining a first object model and a corresponding function that approximates variations to the first object model, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding function that approximates variations to the respective second object model;

determining an approximate location of the object in the image, based on the first object model and its corresponding function; and

refining the location of the object in the image by determining a location of a portion of the object, based on the at least one second object model and its corresponding function.

28. A non-transitory computer-readable medium comprising computer-executable instructions, that when executed perform the method of:

storing a representation of the object, the representation including data defining a first object model and a corresponding function that approximates variations to the first object model, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding function that approximates variations to the respective second object model;

determining an approximate location of the object in the image, based on the first object model and its corresponding function; and

refining the location of the object in the image by determining a location of a portion of the object, based on the at least one second object model and its corresponding function.