AUGMENTED REALITY METHOD APPLIED TO THE INTEGRATION OF A PAIR OF SPECTACLES INTO AN IMAGE OF A FACE

• US 20120313955A1
• Filed: 01/18/2011
• Published: 12/13/2012
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: FittingBox S.A.
• Original Assignee: FittingBox S.A.
• Inventors: Choukroun, Ariel
• Granted Patent: US 9,076,209 B2
• Time in Patent Office: Days
• US Class Current: 345/582
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02C 13/003 Measuring during assembly or fitting of spectacles
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Method for creating a final real-time photorealistic image of a virtual object, corresponding to a real object arranged on an original photo of a user, in a realistic orientation related to the user'"'"'s position, includes: detecting the presence of an area for the object in the photo; determining the position of characteristic points of the area for the object in the photo; determining the 3D orientation of the face, the angles Φ and Ψ of the camera having taken the photo relative to the principal plane of the area; selecting the texture to be used for the virtual object, in accordance with the angle-of-view, and generating the view of the virtual object in 3D; creating a first layered rendering in the correct position consistent with the position of the placement area for the object in the original photo; obtaining the photorealistic rendering by adding overlays to obtain the final image.

Claims

1-24. -24. (canceled)

25. A method for creating a final real-time photorealistic image of a virtual object, corresponding to a real object, which is arranged on an original photo of a user, in a realistic orientation related to the position of said user, characterized in that it comprises the following steps:

510;

detecting the presence of a placement area for the object in an original photo,

530;

determining the position of characteristic points of the placement area for the object in the original photo, step 530 consisting of;

determining a similarity β

, to be applied to an original photo, to obtain a face similar to a reference face in magnification and orientation, and

determining the position of the precise exterior corner A and the precise interior point B for each eye in the face of the original photo.

540;

determining the 3D orientation of the face, i.e. the angles Φ and

Ψ

of the camera having taken the photo relative to the principal plane of the placement area for the object.

26. The method according to claim 25, wherein said method also comprises the following steps:

550;

selecting the texture to be used for the virtual object, in accordance with the angle-of-view, and generating the view of the virtual object in the 3D (Φ

, ψ

)/2D (Θ

, s) position in question,

560;

creating a first rendering by establishing a layered rendering in the correct position consistent with the position of the placement area for the object in the original photo,

570;

obtaining the photorealistic rendering by adding overlays, referred to as semantic overlays, so as to obtain the final image.

27. The method according to claim 25, wherein the object is a pair of spectacles and the placement area is the user'"'"'s face.

28. The method according to claim 25, wherein step 510 uses a first boosting algorithm AD1 trained to determine whether the original photo contains a face.

29. The method according to claim 25, wherein step 530 uses an iterative algorithm that makes it possible to refine the value of the similarity β

and the positions of the characteristic points;

defining the first parameters of similarity β

₀=(tx₀, ty₀, s₀, Θ

₀),

characterizing the eyes in the original photo of the user, from a predefined set of models of eyes DB<sub>models—

eyes</sub> and evaluating the scale,

re-evaluating the parameters of similarity β

₁=(tx₁, ty₁, s₁, Θ

₁).

30. The method according to claim 25, wherein step 530 uses a second boosting algorithm trained with an eyes learning database, comprising a set of positive examples of eyes and a set of negative examples of eyes.

31. The method according to claim 26, wherein step 550 consists of:

1) determining a simplified geometric model of the real pair of spectacles, said model comprising a predefined number N of surfaces and their normals, taking as the orientation of these normals the exterior of the envelop convex to the real pair of spectacles,

2) applying to it, from a predefined set of reference orientations, an orientation closest to angles Φ and

Ψ

,

3) calculating a texture of the simplified geometric model, positioned in the 3D orientation of the reference orientation closest to angles Φ and

Ψ

, using the texture of this reference orientation;

this is equivalent to texturing each of the N surfaces of the simplified geometric model while classifying the surface in the current view into three classifications;

interior surface of the frame, exterior frame of the frame, lens.

32. The method according to claim 31, wherein the simplified geometric model of a real pair of spectacles, consisting of a frame and lenses, is obtained in a phase 100 in which:

a set of shots of the real pair of spectacles to be modeled is produced, with different angles-of-view and using different screen backgrounds with and without the real pair of spectacles,

the simplified geometric model is constructed, consisting of a number N of surfaces surface_j and their normal {right arrow over (n)}_j beginning with a not very dense surface mesh and using an optimization algorithm that deforms the model'"'"'s mesh so that the projections of its silhouette in each of the views best match the silhouettes detected in the images.

33. The method according to claim 32, wherein the number N of surfaces of the simplified geometric model is a value close to twenty.

34. The method according to claim 32, wherein phase 100 also comprises a step 110 consisting of obtaining images of the real pair of spectacles;

the lens must match the lens intended for trying on 500, and that in this step 110;

the real pair of spectacles is photographed at high resolution according to V different reference orientations Orientationⁱ and in N light configurations showing the transmission and reflection of the spectacle lens,

these reference orientations are selected by discretizing a spectrum of orientations corresponding to possible orientations when spectacles are tried on,

V*N high-resolution images of the real pair of spectacles, designated Image-spectacles^i,j, are obtained.

35. The method according to claim 34, wherein the number V of reference orientations is equal to nine, and if an orthogonal reference space with axes x, y, z is defined, where the y-axis corresponds to the vertical axis, Ψ

to the angle of rotation around the x-axis, Φ

to the angle of rotation around the y-axis, the V positions Orientationⁱ selected are such that the angle Ψ

substantially takes the respective values −

16°

, 0°

or 16°

, the angle Φ

takes the respective values −

16°

, 0°

or 16°

.

36. The method according to claim 34, wherein:

the first light configuration respects the colors and materials of the real pair of spectacles, using neutral light conditions;

the V high-resolution transmission images Transmissionⁱ created in this light configuration allow the maximum transmission of light through the lenses to be revealed,

the second light configuration highlights the geometric characteristics of the real pair of spectacles, using conditions of intense reflection;

the V high-resolution reflection images Reflectionⁱ obtained in this second light configuration reveal the physical reflection properties of the lens.

37. The method according to claim 31, wherein phase 100 comprises a step 120 of creating a texture overlay of the frame Frameⁱ, for each of the V reference orientations.

38. The method according to claim 37, wherein in this step 120:

for each of the V reference orientations, the high-resolution reflection image Reflection) is taken,

a binary image is generated with the same resolution as the high-resolution reflection image of the reference orientations;

said binary image is called the lens silhouette Lensⁱ_binary. in this lens silhouette Lensⁱ_binary, the value of the pixel is equal to one if the pixel represents the lenses and zero otherwise.

39. The method according to claim 38, wherein the shape of the lenses needed to generate the lens silhouette Lensⁱ_binary is extracted using an active contours algorithm based on the assumption that the frame and the lenses have different transparencies.

40. The method according to claim 38, wherein, in step 120:

a lens overlay Lens_overlay is generated for each of the reference orientations by copying, for each pixel with a value equal to one in the binary overlay of the lens Lens i_binary, the information contained in the high-resolution reflection image and assigning zero to the other pixels,

this lens overlay Lensⁱ_overlay is a high-definition cropped image of the lens using, for cropping the original high-definition image, the lens silhouette Lensⁱ_binary.

the associated high-resolution reflection image Reflectionⁱ is selected for each of the reference orientations, and a binary background image Backgroundⁱ_binary is generated by automatically extracting the background,

a binary image is generated from the binary overlay of the frame Frameⁱ_binary, by deducting from a neutral image the outline image of the lenses and the outline image of the background,

a texture overlay of the frame behind the lens Frameⁱ<sub>behind—

lens</sub>, with the texture of the frame corresponding to the portion of the frame located behind the lenses, is generated for each of the reference orientations by copying, for each pixel with a value equal to one in the binary lens overlay Lensⁱ_binary, the information contained in the high-resolution transmission image Transmissionⁱ, and assigning zero to the other pixels,

a texture overlay of the frame outside the lens Frameⁱ<sub>exterior—

lens</sub> is generated by copying, for each pixel with a value equal to one in the binary frame overlay Frameⁱ_binary the information contained in the high-resolution reflection image, and assigning zero to the other pixels,

an overlay of the texture of the frame Frameⁱ is defined as the sum of the overlay of the texture of the frame behind the lens Frameⁱ<sub>behind—

lens</sub> and the overlay of the texture of the frame outside the lens Frameⁱ<sub>exterior—

lens</sub>.

41. The method according to claim 37, wherein, in step 550, the texture calculation is performed using overlays associated to the reference orientation closest to angles Φ

and Ψ

, by the following sub-steps;

inversion of the normals {right arrow over (n)}_j of each of the surfaces of the pair of spectacles modeled surface_j and projection of the frame overlay Frameⁱ, limited to the lens space of the reference orientation closest to angles Φ and

Ψ

, to obtain a texture overlay of the internal surface of the frame TextureFrameⁱ<sub>surface—

interior</sub>. that makes it possible to structure the arms of the frame seen through the lens, in a textured reference model, oriented according to the reference orientation closest to angles Φ and

Ψ

,

projection of the frame overlay Frameⁱ, limited to the space outside the lens of the reference orientation closest to angles Φ and

Ψ

, to obtain a texture overlay of the external surface of the frame TextureFrameⁱ<sub>surface—

exterior</sub> that makes it possible to structure the surfaces of the frame outside the lens, in the textured reference model, oriented according to the reference orientation closest to angles Φ and

Ψ

,

projection of the lens overlay limited to the lens to obtain a lens texture overlay TextureLensⁱ that makes it possible to structure the lens, in the textured reference model, oriented according to the reference orientation closest to angles Φ and

Ψ

.

42. The method according to claim 27, wherein step 560 consists of generating an oriented textured model, oriented according to angles Φ

and Ψ and

according to the scale and orientation of the original photo, from a textured reference model, oriented according to the reference orientation closest to angles Φ and

Ψ

, and parameters of similarity β

; and

in that this step comprises the following sub-steps;

using a bilinear affine interpolation to orient an interpolated textured model according to the angles Φ and

Ψ

based on the textured reference model oriented according to the reference orientation closest to these angles Φ and

Ψ

,

using the similarity β

to be applied, so as to obtain the same scale, the same image orientation and the same centering as the original photo, thus producing an oriented textured model.

43. The method according to claim 42, wherein step 560 also comprises a sub-step of geometrically varying the arms of the virtual spectacles according to the morphology of the face of the original photo, so as to obtain a spectacles overlay Spectacles_overlay of the virtual pair of spectacles and a binary overlay Spectacles_oveflay—

_binary, oriented as the original photo, and which can therefore be superimposed on it.

44. The method according to claim 27, wherein step 570 consists of taking into account the light interactions due to wearing the virtual spectacles, particularly taking into account the shadows cast onto the face, the visibility of the skin through the lens of the spectacles, the reflection of the environment on the spectacles.

45. The method according to claim 44, wherein step 570 comprises the following sub-steps:

1) creating a shadow map Visibilityⁱ for each reference orientation, obtained by calculating the light occlusion produced by the real pair of spectacles on each area of the average face when the entire face is lit by a light source, said light source being modeled by a set of point sources emitting in all directions, located at regular intervals in a rectangle,

2) multiplying the shadow map and the photo to obtain a shadowed photo overlay, designated L<sub>skin—

shadowed</sub>,

3) blending the shadowed photo overlay L<sub>skin—

shadowed</sub> and the spectacles overlay Spectacles_overlay by linear interpolation, depending on the coefficient of opacity α

of the lens in an area limited to the binary overlay Spectacles<sub>overlay—

binary</sub> of the virtual pair of spectacles, to obtain a final image;

this is an image of the original photo on which an image of the selected model of spectacles is superimposed, oriented as the original picture and given shadow properties

46. The method according to claim 29, wherein said method comprises in addition a phase 200 of creating a database of models of eyes DB_models—

_eyes, comprising a plurality of photographs of faces referred to as learning photographs App_eyes^k

47. The method according to claim 46, wherein step 200 comprises the following sub-steps:

step 210, of defining a reference face shape and orientation by setting a reference interpupillary distance di₀, by centering the interpupillary segment on the center of the image and orienting this interpupillary segment parallel to the image'"'"'s horizontal axis,

then, for each k^th learning photograph App_eyes^k not yet processed;

step 230, of determining the precise position of characteristic points;

exterior point B_l^k, B_r^k, and interior point A_l^k, A_r^k of each eye and determining the respective geometric center G_l^k, G_r^k of these eyes and the interpupillary distance di^k,

step 231, of transforming this k^th learning photograph App_eyes^k into a gray-scale image App_eyes-gray^k, and normalizing the gray-scale image by applying a similarity S^k(tx, ty, s,) so as to establish the orientation and scale of the reference face to obtain a k^th gray-scale normalized learning photograph App<sub>eyes—

gray—

norm</sub>^k,

step 232, of defining a window of fixed dimensions for each of the two eyes, in the k^th gray-scale normalized learning photograph App<sub>eyes—

gray—

norm</sub>^k;

left patch P_l^k and right patch P_r^k;

the position of a patch P is defined by the fixed distance Δ

between the exterior point of the eye B and the edge of the patch P le closest to this exterior point of the eye B

step 233, for each of the two patches P_l^k, P_r^k associated to the k^th gray-scale normalized learning photograph App<sub>eyes—

gray—

norm</sub>^k of normalizing the gray-scales,

step 234, for the first learning photograph App_eyes¹ of storing each of the patches P_l¹, P_r¹ called descriptor patches, in the eyes database DB<sub>models—

eyes</sub>,

step 235, for each of the patches P associated to the k^th gray-scale normalized learning photograph App<sub>eyes—

gray—

norm</sub>^k of correlating the corresponding normalized texture column-vector T0 with each of the normalized texture column-vectors T0_i of the corresponding descriptor patches,

step 236 of comparing, for each of the patches P_l^k, P_r^k, this correlation measurement with a previously defined correlation threshold threshold, and, if the correlation is less than the threshold, of storing patch P as a descriptor patch in the eyes database DB<sub>models—

eyes</sub>.

48. The method according to claim 47, wherein, in step 232, the fixed distance Δ

is chosen so that no texture exterior to the face is included in patch P, and the width w and height h of patches P_l^k, P_r^k are constant and predefined, so that patch P contains the eye corresponding to this patch P in full, and contains no texture that is exterior to the face, irrespective of the learning photograph App_eyes^k.