System and method for motion capture

• US 10,092,220 B2
• Filed: 03/20/2014
• Issued: 10/09/2018
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: Telecom Italia S.p.A.
• Original Assignee: Telecom Italia S.p.A.
• Inventors: Mauro, Alessandro, Azzaro, Corrado, Albani, Giovanni, Ferraris, Claudia, Nerino, Roberto, Chimienti, Antonio, Pettiti, Giuseppe, Contin, Laura
• Primary Examiner(s): Philippe, Gims S
• Application Number:
  US15/126,904
• Publication Number:
  US 20170086712A1
• Time in Patent Office: 1,664 Days
• Field of Search: 348 77, 348159, 382103, 600473, 600483, 345156, 345420
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2560/0233 Optical standards
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1032 Determining colour for diagnostic purposes measuring colour in general G01J3/46
• CPC Class Codes: A61B 5/1101 Detecting tremor
• CPC Class Codes: A61B 5/1125 Grasping motions of hands
• CPC Class Codes: A61B 5/1127 using markers
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4082 Diagnosing or monitoring movement diseases, e.g. Parkinson, Huntington or Tourette
• CPC Class Codes: A61B 5/7221 Determining signal validity, reliability or quality A61B5/7207 and A61B5/7217 take precedence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A motion capture system includes a wearable device (e.g. a glove) suitable for being worn by a user and including one or more markers having respective colors. A video camera acquires a sequence of color frames of the user moving while wearing the glove, while a range camera acquires corresponding depth frames of the same scene. A processing unit processes both color frames and depth frames for reconstructing the 3D positions of the markers. In particular, the depth information provided by the depth frames are used for isolating a validity area within the color frames, and the markers are searched based on their colors, exclusively within the validity area of the color frames. The movements of the user are then captured as a sequence of positions of the markers. Combined use of color information and depth information provide a very reliable and accurate motion capture.

Claims

1. A motion capture system comprising:

a wearable device configured to be fitted on at least a part of a body of a user, said wearable device comprising at least one marker having a predetermined marker color;

a video camera configured to acquire at least one color frame of a scene comprising said wearable device, said at least one marker being visible in said at least one color frame;

a range camera configured to acquire at least one depth frame of said scene; and

a processing unit configured to receive said at least one color frame from said video camera and said at least one depth frame from said range camera, to process said at least one depth frame for identifying in said at least one color frame a validity area comprising pixels representing said wearable device, to search said at least one marker in said at least one color frame based on said marker color, said search being confined to said validity area, and to capture a motion of said part of the body of said user based on a sequence of positions of said at least one marker.

2. The system according to claim 1, wherein said wearable device is made of a flexible and opaque material.

3. The system according to claim 1, wherein said wearable device has an external surface having a background color different from said marker color.

4. The system according to claim 1, wherein said wearable device comprises at least two markers, said at least two markers comprising a color calibration marker whose marker color is white.

5. The system according to claim 1, wherein said processing unit is configured to perform a brightness calibration, before searching said at least one marker in said at least one color frame, said brightness calibration comprising adjusting a gain of said video camera so as to bring an average brightness of said at least one color frame within a predefined range.

6. The system according to claim 4, wherein said processing unit is configured to identify said validity area in said at least one color frame by:

identifying said color calibration marker in said at least one color frame, based on its shape;

determining a two-dimensional position of a center of said color calibration marker in said at least one color frame and a depth of said center of said color calibration marker in said at least one depth frame;

identifying in said at least one depth frame a cluster of pixels whose depth is substantially the same as said depth of said center of said color calibration marker and determining a centroid of said cluster;

constructing a segmentation solid around said centroid of said cluster, said segmentation solid having shape and size suitable for containing said wearable device when fitted on said at least a part of said body of said user, independently of a current position of said at least a part of said body of said user; and

identifying said validity area in said at least one color frame as a portion of said color frame formed by pixels included in said segmentation solid.

7. The system according to claim 1, wherein said processing unit is configured to perform a color calibration, before searching said at least one marker in said at least one color frame, said color calibration comprising one or more of calculating one or more color correction factors to be applied to said at least one color frame and selecting a set of color thresholds to be used for searching said at least one marker in said validity area of said at least one color frame.

8. The system according to claim 7, wherein said processing unit is configured to calculate said one or more color correction factors by:

identifying a color calibration marker in said at least one color frame, based on its shape;

calculating average color components of a portion of said color calibration marker; and

calculating said one or more color correction factors to be applied to said at least one color frame based on said average color components of said portion of said color calibration marker.

9. The system according to claim 7, wherein said processing unit is configured to select said set of color thresholds to be used for searching said at least one marker in said validity area of said at least one color frame by:

identifying said calibration marker in said color frame, based on its shape;

calculating an average brightness of a portion of said color calibration marker; and

selecting said set of color thresholds based on said average brightness of said portion of said color calibration marker.

10. The system according to claim 7, wherein said processing unit is configured to apply said one or more color correction factors only to said validity area of said at least one color frame, before searching said at least one marker in said validity area of said at least one color frame based on said marker color.

11. The system according to claim 1, wherein said processing unit is configured to search said at least one marker in said validity area of said at least one color frame by identifying, within said validity area of said at least one color frame, at least one marker color blob formed by contiguous pixels having said marker color.

12. The system according to claim 11, wherein said processing unit is further configured to determine a three-dimensional position of said at least one marker by processing both said at least one color frame and said at least one depth frame.

13. The system according to claim 12, wherein said processing unit is configured to determine said three-dimensional position of said at least one marker by:

determining a two-dimensional position of a center of said at least one marker color blob in said at least one color frame;

determining a pseudo-three dimensional position of said at least one marker, said pseudo-three dimensional position comprising said two-dimensional position of said center of said at least one marker color blob in said at least one color frame and a depth of said center of said at least one marker color blob in said at least one depth frame; and

converting said pseudo-three dimensional position of said at least one marker into a tern of coordinates that indicate said three-dimensional position of said at least one marker relative to a Cartesian coordinate system.

14. The system according to claim 3, wherein said processing unit is further configured to refine said validity area of said at least one color frame using color information.

15. The system according to claim 14, wherein said processing unit is configured to refine said validity area of said at least one color frame by:

identifying, within said validity area of said at least one color frame, at least one marker color blob formed by contiguous pixels having said marker color;

identifying, within said validity area of said at least one color frame, a background color blob formed by contiguous pixels having said background color;

calculating a separate color histogram for each one of said at least one marker color blob and said background color blob;

calculating a cumulative color histogram by merging said separate color histograms for said at least one marker color blob and said background color blob; and

refining said validity area of said at least one color frame by excluding from said validity area those pixels whose probability of belonging to said cumulative color histogram is substantially null.

16. A method for capturing motion of at least part of a body of a user upon which a wearable device is fitted, said wearable device comprising at least one marker having a predetermined marker color, said method comprising:

a) receiving, from a video camera, at least one color frame of a scene comprising said wearable device, said at least one marker being visible in said at least one color frame;

b) receiving, from a range camera, at least one depth frame of said scene;

c) processing said at least one depth frame for identifying in said at least one color frame a validity area comprising pixels representing said wearable device;

d) searching said at least one marker in said at least one color frame based on said marker color, said search being confined to said validity area; and

e) capturing a motion of said part of the body of said user based on a sequence of positions of said at least one marker.

17. A computer readable medium including software code portions stored thereon that, when executed by a computer, perform a method for capturing motion of at least part of a body of a user upon which a wearable device comprising at least one marker having a predetermined marker color is fitted, the method comprising the steps of:

a) receiving, from a video camera, at least one color frame of a scene comprising said wearable device, said at least one marker being visible in said at least one color frame;

b) receiving, from a range camera, at least one depth frame of said scene;

c) processing said at least one depth frame for identifying in said at least one color frame a validity area comprising pixels representing said wearable device;

d) searching said at least one marker in said at least one color frame based on said marker color, said search being confined to said validity area; and

e) capturing a motion of said part of the body of said user based on a sequence of positions of said at least one marker.