Computer-implemented method and apparatus for tracking and reshaping a human shaped figure in a digital world video

• US 9,191,579 B2
• Filed: 11/29/2011
• Issued: 11/17/2015
• Est. Priority Date: 11/29/2010
• Status: Active Grant

• Current Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Original Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Inventors: Seidel, Hans-Peter, Theobalt, Christian, Thormaehlen, Thorsten, Jain, Arjun
• Primary Examiner(s): Zhao, Daquan
• Application Number:
  US13/823,813
• Publication Number:
  US 20130330060A1
• Time in Patent Office: 1,449 Days
• Field of Search: 345/473, 345/474, 345/679, 345/380, 345/681, 386/278, 386/280, 386/282
• US Class Current: 1/1
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

The invention concerns a computer-implemented method for tracking and reshaping a human-shaped figure in a digital video comprising the steps: acquiring a body model of the figure from the digital video, adapting a shape of the body model, modifying frames of the digital video, based on the adapted body model and outputting the digital video.

Claims

1. Computer-implemented method for tracking and reshaping a human-shaped figure in a digital video comprising the steps:

acquiring a body model of the figure from the digital video;

adapting a shape of the body model;

modifying frames of the digital video, based on the adapted body model; and

outputting the digital video,

wherein pose and shape parameters of the model are acquired by measuring the misalignment of a silhouette boundary of a re-projected model with a silhouette boundary of the figure, and

wherein acquiring pose and shape parameters further comprises minimizing a misalignment between the figure in a video frame and the body model re-projected onto the video frame.

2. A method according to claim 1, wherein the body model comprises a kinematic skeleton model and a surface mesh.

3. A method according to claim 1, wherein the body model uses a normal skinning approach for modeling pose-dependent surface mesh adaptation.

4. A method according to claim 1, further comprising the step of providing an interface in which a user may explicitly mark additional image points to be tracked and in which broken trajectories may be linked.

5. A method according to claim 1, wherein the motion of the model is tracked in real-time from the video and/or depth data provided by an additional sensor.

6. A method according to claim 1, wherein the digital video is output in real time.

7. Use of a method according to claim 1 for visualizing a human person in a different shape.

8. Use of a method according to claim 1 for warping images of a human person, based on an image of a different human person.

9. A non-transitory computer-readable medium, storing instructions that, when executed, implement a method according to claim 1.

11. A method of claim 1, wherein the body model is acquired using the digital video and additional sensor equipment.

12. A method according to claim 11, wherein the additional sensor equipment is a depth camera or another scanner or device for capturing 2D or 3D static or time-varying shape, either online or off-line.

13. A method according to claim 12, wherein the additional sensor equipment is an optical motion capture system, an electromagnetic motion capture, any form of motion capture suit or an exoskeleton for pose estimation.

10. A computer-implemented method for tracking and reshaping a human-shaped figure in a digital video, the method comprising:

providing an interface for setting shape attributes of a body model;

wherein the shape attributes of the body model comprise at least one of height, weight, breast girth, waist girth, hips girth, leg length and muscularity;

acquiring a body model of the figure from the digital video;

adapting a shape of the body model, wherein the shape of the body model is adapted by mapping shape attributes set via the interface to the shape;

modifying frames of the digital video, based on the adapted body model; and

outputting the digital video.

14. The method according to claim 10, further comprising:

providing an interface in which a user may explicitly mark additional image points to be tracked and in which broken trajectories may be linked.

15. The method according to claim 10, wherein pose and shape parameters of the model are acquired by measuring a misalignment of a silhouette boundary of the re-projected model with a silhouette boundary of the figure.

16. The method of claim 15 wherein acquiring pose and shape parameters further comprises minimizing a misalignment between the figure in a video frame and the body model re-projected onto the video frame.