Visual and physical motion sensing for three-dimensional motion capture

• US 9,142,024 B2
• Filed: 07/21/2009
• Issued: 09/22/2015
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Lucasfilm Limited (The Walt Disney Company)
• Original Assignee: Lucasfilm Limited (The Walt Disney Company)
• Inventors: Sullivan, Steve, Wooley, Kevin, Allen, Brett A., Sanders, Michael
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): Sadio, Insa
• Application Number:
  US12/506,798
• Publication Number:
  US 20100164862A1
• Time in Patent Office: 2,254 Days
• Field of Search: 345156-184, 348/169
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system includes a visual data collector for collecting visual information from an image of one or more features of an object. The system also includes a physical data collector for collecting sensor information provided by at one or more sensors attached to the object. The system also includes a computer system that includes a motion data combiner for combining the visual information the sensor information. The motion data combiner is configured to determine the position of a representation of one or more of the feature in a virtual representation of the object from the combined visual information and sensor information. Various types of virtual representations may be provided from the combined information, for example, one or more poses (e.g., position and orientation) of the object may be represented.

Claims

1. A computer-implemented method comprising:

obtaining visual information from a first image of an object, the object comprising at least a first feature and a second feature, each feature having a respective visual marker and a respective inertial sensor attached, the first feature being moveable with respect to the second feature;

receiving first position-related information that corresponds to the first image and that is wirelessly transmitted from at least one of the inertial sensors attached to the first and second features of the object;

generating an estimated pose of the object using the first position-related information; and

generating a refined pose of the object using the estimated pose and the visual information, wherein the refined pose of the object comprises a position of the first feature and a position of the second feature in a virtual representation of the object and wherein generating the refined pose comprises using the estimated pose to provide at least one location to detect at least one of the visual markers.

2. The computer-implemented method of claim 1, further comprising:

obtaining second position-related information that corresponds to a second image and that is wirelessly transmitted from at least one of the inertial sensors; and

estimating a position of the first feature in the second image based upon the second position-related information.

3. The computer-implemented method of claim 2, wherein estimating the position of the first feature in the second image includes determining a region of the second image likely to include an image of the first feature based upon the second position-related information.

4. The computer-implemented method of claim 3, wherein the position of the first feature in the first image and the position of the first feature in the second image are used to track the movement of the first feature.

5. The computer-implemented method of claim 1, further comprising:

determining an initial position of the second feature from the first position-related information.

6. The computer-implemented method of claim 1, wherein the first position-related information from at least one of the inertial sensors represents orientation information local to the inertial sensor.

7. The computer-implemented method of claim 1, wherein the visual information includes global position information.

8. The computer-implemented method of claim 1, wherein the object is capable of covering a portion of a body.

9. The computer-implemented method of claim 1, wherein the visual information is obtained from a camera, and wherein third position-related information is received from at least one inertial sensor attached to the camera.

10. The computer-implemented method of claim 1, wherein combining the visual information and the first position-related information includes synchronizing the visual information and the first position-related information.

11. The computer-implemented method of claim 1, wherein the virtual representation of the object is a character of a video game.

12. The computer-implemented method of claim 1, wherein the first and second position-related information are part of a time series of data wirelessly transmitted by the inertial sensors.

13. The computer-implemented method of claim 1, wherein the visual information includes a silhouette of the object.

14. The computer-implemented method of claim 1, wherein the first feature is a naturally occurring feature.

15. The computer-implemented method of claim 1, wherein the first feature is a synthetic feature.

16. The computer-implemented method of claim 1, wherein at least one of the visual information and the first position-related information is used to calibrate the virtual representation of the object.

17. The method of claim 1, further comprising using the generated refined pose of the object with the first position-related information to generate an updated estimated pose of the object, and using the updated estimated pose of the object with the visual information from the first image of the object to produce an updated refined pose of the object corresponding to the first image of the object.

18. A system, comprising:

a visual data collector capable of obtaining visual information from a first image of an object, the object comprising at least a first feature and a second feature, each feature having a respective visual marker and a respective inertial sensor attached, the first feature being moveable with respect to the second feature;

a physical data collector capable of receiving first position-related information that corresponds to the first image and that is wirelessly transmitted by at least one of the respective inertial sensors attached to the first and second features of the object; and

a computer system including a motion data combiner capable of combining the visual information and the position-related information, wherein the motion data combiner is configured to execute a method comprising;

generating an estimated pose of the object, using the first position-related information;

generating a refined pose of the object using the estimated pose and the visual information, the refined pose of the object comprising a position of the first feature and a position of the second feature in a virtual representation of the object and wherein generating the refined pose comprises using the estimated pose to provide at least one location to detect at least one of the visual markers.

19. The system of claim 18, wherein the motion data combiner is further configured to:

obtain second position-related information that corresponds to a second image and that is wirelessly transmitted from at least one of the inertial sensors; and

estimate the position of the first feature in a second image based upon the second position-related information.

20. The system of claim 19, wherein the motion data combiner is configured to estimate a region of the second image likely to include a representation of the first feature based upon the position-related information.

21. The system of claim 19, wherein the motion data combiner is configured to track the movement of the first feature based upon the position of the first feature in the first image and the position of the first feature in the second image.

22. The system of claim 18, wherein the motion data combiner is configured to determine an initial position of the second feature from the first position-related information.

23. The system of claim 18, wherein the object is capable of covering a portion of a body.

24. The system of claim 18, wherein the visual information is obtained from a camera, and wherein third position-related information is received from at least one inertial sensor attached to the camera.

25. The system of claim 18, wherein the motion data combiner is configured to synchronize the visual information and the first position-related information to combine the visual information and the first position-related information.

26. The system of claim 18, wherein the first and second position-related information are part of a time series of data provided by the inertial sensors.

27. The system of claim 18, further comprising using the generated refined pose of the object with the first position-related information to generate an updated estimated pose of the object, and using the updated estimated pose of the object with the visual information from the first image of the object to produce an updated refined pose of the object corresponding to the first image of the object.

28. A non-transitory computer program product tangibly embodied in an non-transitory information carrier and comprising instructions that when executed by a processor perform a method comprising:

obtaining visual information from a first image of an object, the object comprising at least a first feature and a second feature, each feature having a respective visual marker and a respective inertial sensor attached, the first feature being moveable with respect to the second feature;

receiving first position-related information that corresponds to the first image and that is wirelessly transmitted from at least one sensor attached to the object;

generating an estimated pose of the object using the first position-related information;

generating a refined pose of the object using the estimated pose and the visual information, wherein the refined pose of the object comprises a position of the first feature and a position of the second feature in a virtual representation of the object and wherein generating the refined pose comprises using the estimated pose to provide at least one location to detect at least one of the visual markers.

29. The computer program product of claim 28, wherein the method further comprises:

obtaining second position-related information that corresponds to a second image and that is wirelessly transmitted from at least one of the inertial sensors; and

estimating a position of the first feature in the second image based upon the second position-related information.

30. The computer program product of claim 29, wherein estimating the position of the first feature in the second image includes determining a region of the second image likely to include a representation of the first feature based upon the second position-related information.

31. The computer program product of claim 30, wherein the position of the first feature in the first image and the position of the first feature in the second image are used to track the movement of the first feature.

32. The computer program product of claim 28, wherein the method further comprises:

determining an initial position of the second feature from the first position-related information.

33. The computer program product of claim 28, wherein the first position-related information from at least one of the inertial sensors represents orientation information local to the inertial sensor.

34. The computer program product of claim 28, wherein the visual information includes global position information.

35. The computer program product of claim 28, wherein the object is capable of covering a portion of a body.

36. The computer program product of claim 28, wherein the visual information is obtained from a camera, and wherein third position-related information is received from at least one inertial sensor attached to the camera.

37. The computer program product of claim 28, wherein combining the visual information and the first position-related information includes synchronizing the visual information and the first position-related information.

38. The computer program product of claim 28, wherein the virtual representation of the object is a character of a video game.

39. The computer program product of claim 28, wherein the first and second position-related information are part of a time series of data wirelessly transmitted by the inertial sensors.

40. The non-transitory computer program product of claim 28, further comprising using the generated refined pose of the object with the first position-related information to generate an updated estimated pose of the object, and using the updated estimated pose of the object with the visual information from the first image of the object to produce an updated pose of the object corresponding to the first image of the object.

41. A system comprising:

a visual data collector;

a physical data collector; and

a computer system including,

a memory, and

a processor capable of executing instructions that perform a method comprising;

collecting by the visual data collector visual information from a plurality of images, wherein each image includes content that represents a plurality of visual markers attached to at least a first feature and a second feature of a body suit, the first feature being moveable with respect to the second feature, and each feature having attached a respective one of the plurality of visual markers;

receiving by the physical data collector first position-related information that corresponds to a first of the plurality of images, and that is wirelessly transmitted by a plurality of inertial sensors also attached to the body suit, wherein a first inertial sensor is attached to the first feature and a second inertial sensor is attached to the second feature;

generating an estimated pose of the body suit using the first position-related information;

generating a refined pose of the body suit using the estimated pose and the visual information, wherein the refined pose of the object comprises a position of the first feature and a position of the second feature in a virtual representation of the object and wherein generating the refined pose comprises using the estimated pose to provide at least one location to detect at least one of the visual markers.

42. The system of claim 41, wherein the method further comprises:

tracking the movement of the plurality of visual markers across the plurality of images.

43. The system of claim 41, further comprising determining respective positions of the representations of the plurality of visual markers in the first of the plurality of images by determining a region in the first of the plurality of images that is likely to include a representation of at least one of the plurality of visual markers based upon the first position-related information provided by the plurality of inertial sensors.

44. The system of claim 41, wherein the method executed by the processor further comprises:

determining initial positions for each of the plurality of visual markers based upon the first position-related information.

45. The system of claim 41, wherein the visual data collector further comprises:

one or more cameras for collecting the plurality of images, wherein one or more inertial sensors are attached to each of the one or more cameras for collecting calibration information.

46. The system of claim 41, wherein the visual data collector further comprises:

one or more cameras for collecting the plurality of images, wherein the first position-related information is used to calibrate the position of at least one of the cameras.

47. The system of claim 41, wherein the method executed by the processor further comprises:

producing a series of poses of the virtual representation of the body suit corresponding to the plurality of images based upon the combined visual information and position-related information.

48. The system of claim 41, further using the generated refined pose of the object with the first position-related information to generate an updated estimated pose of the object, and using the updated estimated pose of the object with the visual information from the first image of the object to produce an updated refined pose of the object corresponding to the first image of the object.