Method and arrangement for 3-dimensional image model adaptation

• US 9,269,194 B2
• Filed: 06/04/2012
• Issued: 02/23/2016
• Est. Priority Date: 06/20/2011
• Status: Expired due to Fees

• Current Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Original Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Inventors: Tytgat, Donny, Lievens, Sammy, Aerts, Maarten
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US14/122,165
• Publication Number:
  US 20140212031A1
• Time in Patent Office: 1,359 Days
• Field of Search: 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models

Abstract

Method for adapting a 3D model (m) of an object, said method comprising the steps of

• • performing at least one projection of said 3D model to obtain at least one 2D image model projection (p1) with associated depth information (d1),
  • performing at least one state extraction operation on said at least one 2D image model projection (p1), thereby obtaining at least one state (s1)
  • adapting said at least one 2D image model projection (p1) and said associated depth information (d1) in accordance with said at least one state (s1) and with a target state (s), thereby obtaining at least one adapted 2D image model (p1′) and an associated adapted depth (d1′)
  • back-projecting said at least one adapted 2D image model (p1′) to 3D, based on said associated adapted depth (d1′) for thereby obtaining an adapted 3D model (m′).

Claims

1. A method for adapting a 3D model of an object, comprising the steps of:

performing, via a processor, at least one projection of the 3D model to obtain at least one 2D image model projection with associated depth information;

performing, via the processor, at least one state extraction operation on the at least one 2D image model projection to obtain at least one state comprising a configuration of object features, the features being represented by a set of values;

either receiving, via the processor, external state information being offline determined state information based on 2D still image data or on descriptive information,

or receiving, via the processor, an external 2D image input, from which the external state information is extracted;

determining, via the processor, a target state from the at least one state and from the external state information by performing a weighted combination of the at least one state and the external state information with the weights reflecting confidences of the states determined during the state extraction;

adapting, via the processor, the at least one 2D image model projection and the associated depth information using the at least one state and the target state to obtain at least one adapted 2D image model and an associated adapted depth; and

back-projecting, via the processor, the at least one adapted 2D image model to 3D based on the associated adapted depth to obtain an adapted 3D model.

2. The method of claim 1, wherein the adapted 3D model is further determined based on 3D model information.

3. The method of claim 2, wherein the external state information comprises high level semantic information with respect to the object features.

4. The method of claim 1, wherein the step of adapting comprises determining, via the processor, a new location for at least one object feature of the at least one state based on a corresponding object feature of the state of the external image input.

5. The method of claim 1, wherein at least one of the 2D projections of the 3D model is performed based on a virtual camera deduced from the external image input.

12. The method of claim 1 wherein the adapting the at least one 2D image model projection and the associated depth information using the at least one state and the target state is performed on features extracted from external live video and projected 2D images, and wherein new positions of the features for the projected 2D images are determined based on locations of the features of the live video.

6. An apparatus comprising one or more processors configured to:

perform at least one projection of the 3D model to obtain at least one 2D image model projection with associated depth information;

perform at least one state extraction operation on the at least one 2D image model projection to obtain at least one state comprising a configuration of object features, the features being represented by a set of values;

either receive external state information being offline determined state information based on 2D still image data or on descriptive information,

or receive an external 2D image input, from which the external state information is extracted;

determine a target state from the at least one state and from the external state information by performing a weighted combination of the at least one state and the external state information with the weights reflecting confidences of the states determined during the state extraction;

adapt the at least one 2D image model projection and the associated depth information using the at least one state and the target state to obtain at least one adapted 2D image model and an associated adapted depth; and

back-project the at least one adapted 2D image model to 3D based on the associated adapted depth to obtain an adapted 3D model.

7. The apparatus of claim 6, wherein the adapted 3D model is further determined based on 3D model information.

8. The apparatus of claim 6, wherein the external state information comprises high level semantic information with respect to the object features.

9. The apparatus of claim 6, wherein the adapt the at least one 2D image model projection further comprises one or more processors configured to determine a new location for at least one object feature of the at least one state based on a corresponding object feature of the state of the external image input.

13. An image processing apparatus comprising an apparatus as set out in claim 6.

10. A non-transitory computer readable medium encoding a machine-executable program of instructions to perform a method, the method comprising the steps of:

performing at least one projection of the 3D model to obtain at least one 2D image model projection with associated depth information;

performing at least one state extraction operation on the at least one 2D image model projection to obtain at least one state comprising a configuration of object features, the features being represented by a set of values;

either receiving external state information being offline determined state information based on 2D still image data or on descriptive information,

or receiving an external 2D image input, from which the external state information is extracted;

determining a target state from the at least one state and from the external state information by performing a weighted combination of the at least one state and the external state information with the weights reflecting confidences of the states determined during the state extraction;

adapting the at least one 2D image model projection and the associated depth information using the at least one state and the target state to obtain at least one adapted 2D image model and an associated adapted depth; and

back-projecting the at least one adapted 2D image model to 3D based on the associated adapted depth to obtain an adapted 3D model.

11. The non-transitory computer readable medium of claim 10, wherein the step of adapting comprises determining a new location for at least one object feature of the at least one state based on a corresponding object feature of the state of the external image input.

14. An apparatus for adapting a 3D model of an object, comprising one or more processors adapted to:

perform at least one projection of the 3D model to obtain at least one 2D image model projection with associated depth information;

perform at least one state extraction operation on the at least one 2D image model projection to obtain at least one state comprising a configuration of object features, the features being represented by a set of values;

either adapted to receive external state information being offline determined state information based on 2D still image data or on descriptive information,

or adapted to receive an external 2D image input, from which the external state information is extracted;

determine a target state from the at least one state and from the external state information by performing a weighted combination of the at least one state and the external state information with the weights reflecting confidences of the states determined during the state extraction;

adapt the at least one 2D image model projection and the associated depth information using the at least one state and the target state to obtain at least one adapted 2D image model and an associated adapted depth, wherein the adapt the at least one 2D image model projection and the associated depth information using the at least one state and the target state is performed on features extracted from external live video and projected 2D images; and

back-project the at least one adapted 2D image model to 3D based on the associated adapted depth to obtain an adapted 3D model.