Method of hiding an object in an image or video and associated augmented reality process

• US 9,892,561 B2
• Filed: 10/05/2016
• Issued: 02/13/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: FittingBox S.A.
• Original Assignee: FittingBox S.A.
• Inventors: Choukroun, Ariel, Guenard, Jerome
• Primary Examiner(s): Thompson, James A
• Application Number:
  US15/285,554
• Publication Number:
  US 20180005448A1
• Time in Patent Office: 496 Days
• Field of Search: None
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/193 : Preprocessing; Feature extraction

Abstract

A method for generating a final image from an initial image including an object suitable to be worn by an individual. The presence of the object in the initial image is detected. A first layer is superposed on the initial image. The first layer includes a mask at least partially covering the object in the initial image. The appearance of at least one part of the mask is modified. The suppression of all or part of an object in an image or a video is enabled. Also, a process of augmented reality intended to be used by an individual wearing a vision device on the face, and a try-on device for a virtual object.

Claims

1. A method for generating a final image from an initial image comprising a physical object suitable to be worn by an individual, comprising the steps of:

acquiring an image of the individual wearing the physical object on the individual'"'"'s face, said image being the initial image;

detecting a presence of said physical object in the initial image;

generating a mask at least partially covering the physical object in the initial image;

superposing a first layer on the initial image, the first layer including the mask at least partially covering the physical object in the initial image;

generating a texture reproducing element in a background of the physical object to suppress all or part of the image of the physical object in the final image; and

modifying an appearance of at least one part of the mask by applying to the mask the generated texture.

2. The image generation method according to claim 1, wherein the modification of the appearance of the mask comprises a step of substitution of a texture of all or part of the object in the final image.

3. The image generation method according to claim 1, wherein the mask also covers all or part of a shadow cast by the object.

4. The image generation method according to claim 1, further comprising the steps of superposing a second layer on the initial image over the first layer, the second layer including an element at least partially covering the mask.

5. The image generation method according to claim 1, prior to step of superposing the first layer, further comprising the steps of:

determining an orientation of the object in relation to a capture device acquiring the initial image; and

determining a characteristic dimension of the object in the initial image.

6. The image generation method according to claim 5, prior to step of superposing the first layer, further comprising the steps of:

calculating a three-dimensional model of the object; and

calculating the mask by geometric projection of the three-dimensional model on the first layer, the model having on the first layer the same orientation and the same characteristic dimension as the object.

7. The image generation method according to claim 6, wherein the calculation of the model of the object is performed based on:

at least one image of the object alone;

or

at least an image of the object worn on a face of the individual.

8. The image generation method according to claim 1, wherein the object comprising a frame extending to either side of a face is worn on the face of the individual;

and further comprising a step of identifying the frame from frames previously modeled, the mask having been generated based on the model of the identified frame.

9. The image generation method according to claim 8, wherein the identification of the frame is based on at least one of the following criteria:

support curves generated and adjusted to contours of the frame;

a shape of the frame;

one or more colors of the frame;

one or more textures of the frame; and

a logo presented on the frame.

10. The image generation method according to claim 1, further comprising a step of calculating a representation of an environment of the object.

11. The image generation method according to claim 10, wherein the step of modification of the appearance of the mask comprises sub-steps of:

geometric projecting the representation of the environment on which an intermediate layer is superimposed on the first layer; and

determining a new color of a pixel of the mask according to a color of at least one pixel of an intermediate layer near the pixel of the mask.

12. The image generation method according to claim 10, further comprising a step of detecting a presence of a face in the environment and in that a representation of the environment comprises a model of the detected face to which a texture of the face is applied.

13. The image generation method according to claim 12, further comprising a step of determining an orientation of the face relative to a capture device and in that the model of the detected face is essentially positioned according to an orientation previously established.

14. The image generation method according to claim 12, wherein the mask at least partially covering the object worn on the face is developed based on the geometric projection of the model of the face on the first layer.

15. The image generation method according to claim 12, further comprising the steps of:

estimating a colorimetric transformation from the initial image of at least one pixel of the model of the face; and

converting one or more colors of all or part of the model of the face.

16. The image generation method according to claim 15, wherein a color of a pixel on the texture of the face is determined through an inpainting method based on one of the following criteria:

colors of a patch near the pixel, a position of the patch being essentially located on at least one of a perpendicular and on a vertical of the pixel; and

the model of the face, previously established and oriented, the model of the face including a representation of eyes.

17. The image generation method according to claim 12, further comprising a step of identifying at least one ocular region on the texture of the face, the ocular region corresponding to a position of an eye of the detected face.

18. The image generation method according to claim 17, wherein the ocular region is filled with a knowledge of a topology of the eye of the detected face.

19. The image generation method according to claim 10, wherein the calculation of the representation of the environment of the object worn on a face of the individual is generated without detecting the face in the environment.

20. The image generation method according to claim 10, wherein the calculation of the representation of the environment comprises a sub-step of correction of an optical distortion caused by a transparent element positioned between the environment and a capture device acquiring the initial image.

21. The image generation method according to claim 10, wherein at least one of the representation of the environment and a model of the object are updated based on multiple initial images acquired according to multiple distinct viewing angles.

22. The image generation method according to claim 1 is applied on all or part of a sequence of images forming a video.

23. The image generation method according to claim 22, wherein at least one of a representation of an environment and a model of the object are updated at each image of the sequence.

24. The image generation method according to claim 1, wherein the generation of the final image is generated in real time from the initial image.

25. A method of augmented reality intended to be used by an individual wearing a portable device on a face, comprising the steps of:

acquiring in real time a video of the individual wearing the portable device on the face;

displaying in real time the video in which an appearance of the portable device is entirely or partially modified by the image generation method as claimed in claim 1.

26. The method of augmented reality according to claim 25, wherein the portable device is entirely or partially suppressed in the video displayed in real time.

27. The method of augmented reality according to claim 25, wherein the portable device worn by the individual comprises a frame and corrective lenses adapted to a vision of the individual.

28. The method of augmented reality according to claim 25, wherein the individual wearing the portable device tries on a virtual object superimposed at least in part in the video on the portable device partially or totally suppressed.

29. The method of augmented reality according to claim 25, further comprising a step of initializing the model of the face of the individual based on at least one image of the individual not wearing the portable device on the face;

or on multiple images of the individual wearing the portable device on the face, the images corresponding to different viewing angle of the face.