Visual and Physical Motion Sensing for Three-Dimensional Motion Capture

• US 20100164862A1
• Filed: 07/21/2009
• Published: 07/01/2010
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Lucasfilm Limited (The Walt Disney Company)
• Original Assignee: Lucasfilm Limited (The Walt Disney Company)
• Inventors: Allen, Brett A., Sanders, Michael, Wooley, Kevin, Sullivan, Steve
• Granted Patent: US 9,142,024 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system includes a visual data collector for collecting visual information from an image of one or more features of an object. The system also includes a physical data collector for collecting sensor information provided by at one or more sensors attached to the object. The system also includes a computer system that includes a motion data combiner for combining the visual information the sensor information. The motion data combiner is configured to determine the position of a representation of one or more of the feature in a virtual representation of the object from the combined visual information and sensor information. Various types of virtual representations may be provided from the combined information, for example, one or more poses (e.g., position and orientation) of the object may be represented.

Claims

1. A computer-implemented method comprising:

combining visual information from a first image of at least a first feature of an object and sensor information provided by at least one sensor attached to the object; and

determining the position of a representation of the first feature in a virtual representation of the object from the combined visual information and sensor information.

2. The computer-implemented method of claim 1, further comprising:

determining the position of the first feature in a second image based upon the sensor information provided by the sensor.

3. The computer-implemented method of claim 2, wherein determining the position of the first feature in the second image includes determining a region of the second image likely to include a representation of the first feature based upon the sensor information provided by the sensor.

4. The computer-implemented method of claim 3, wherein the position of the first feature in the first image and the position of the first feature in the second image are used to track the movement of the first feature.

5. The computer-implemented method of claim 1, further comprising:

determining the initial position of at least a second feature from the sensor information provided by the sensor.

6. The computer-implemented method of claim 1, wherein the sensor information from the sensor represents orientation information local to the sensor.

7. The computer-implemented method of claim 1, wherein the visual information represents global position information.

8. The computer-implemented method of claim 1, wherein the virtual representation of the object represents a pose of the object during collection of the visual information and the sensor information.

9. The computer-implemented method of claim 1, wherein the object is capable of covering a portion of a body.

10. The computer-implemented method of claim 1, wherein the object is a camera.

11. The computer-implemented method of claim 1, wherein combining the visual information and the sensor information includes synchronizing the visual information and the sensor information.

12. The computer-implemented method of claim 1, wherein the virtual representation of the object is an online character of a video game.

13. The computer-implemented method of claim 1, wherein the senor is an inertial sensor.

14. The computer-implemented method of claim 1, wherein the sensor information includes a time series of data provided by the sensor.

15. The computer-implemented method of claim 1, wherein the visual information represents a silhouette of the object.

16. The computer-implemented method of claim 1, wherein the first feature is a naturally occurring feature.

17. The computer-implemented method of claim 1, wherein the first feature is a synthetic feature.

18. The computer-implemented method of claim 17, wherein the first feature is a marker attached to the object.

19. The computer-implemented method of claim 1, wherein at least one of the visual information and one of the sensor information is used to calibrate the virtual representation of the object.

20. A system, comprising:

a visual data collector for collecting visual information from a first image of at least a first feature of an object;

a physical data collector for collecting sensor information provided by at least one sensor attached to the object; and

a computer system including a motion data combiner for combining the visual information the sensor information, the motion data combiner is configured to determine the position of a representation of the first feature in a virtual representation of the object from the combined visual information and sensor information.

21. The system of claim 20, wherein the motion data combiner is further configured to determine the position of the first feature in a second image based upon the sensor information provided by the sensor.

22. The system of claim 21, wherein the motion data combiner is configured to determine a region of the second image likely to include a representation of the first feature based upon the sensor information provided by the sensor.

23. The system of claim 20, wherein the motion data combiner is configured to track the movement of the first feature based upon the position of the first feature in the first image and the position of the first feature in the second image.

24. The system of claim 20, wherein the motion data combiner is configured to determine the initial position of a second feature from the sensor information provided by the sensor.

25. The system of claim 20, wherein the motion data combiner uses the visual information to define a pose of the object produced by the sensor information from the sensor.

26. The system of claim 20, wherein the motion data combiner uses the combined visual information and sensor information to produce a pose representative of the object.

27. The system of claim 20, wherein the virtual representation of the object represents a pose of the object during collection of the visual information and the sensor information.

28. The system of claim 20, wherein the object is capable of covering a portion of a body.

29. The system of claim 20, wherein the object is a camera.

30. The system of claim 20, wherein the motion data combiner is configured synchronize the visual information and the sensor information to combine the visual information and the sensor information.

31. The system of claim 20, wherein the senor is an inertial sensor.

32. The system of claim 20, wherein the sensor information includes a time series of data provided by the sensor.

33. A computer program product tangibly embodied in an information carrier and comprising instructions that when executed by a processor perform a method comprising:

combining visual information from a first image of at least a first feature of an object and sensor information provided by at least one sensor attached to the object; and

determining the position of a representation of the first feature in a virtual representation of the object from the combined visual information and sensor information.

34. The computer program product of claim 33, wherein the method further comprises:

determining the position of the first feature in a second image based upon the sensor information provided by the sensor.

35. The computer program product of claim 34, wherein determining the position of the first feature in the second image includes determining a region of the second image likely to include a representation of the first feature based upon the sensor information provided by the sensor.

36. The computer program product of claim 35, wherein the position of the first feature in the first image and the position of the first feature in the second image are used to track the movement of the first feature.

37. The computer program product of claim 33, wherein the method further comprises:

determining the initial position of a second feature from the sensor information provided by the sensor.

38. The computer program product of claim 33, wherein the sensor information from the sensor represents orientation information local to the sensor.

39. The computer program product of claim 33, wherein the visual information represents global position information.

40. The computer program product of claim 33, the virtual representation of the object represents a pose of the object during collection of the visual information and the sensor information.

41. The computer program product of claim 33, wherein the object is capable of covering a portion of a body.

42. The computer program product of claim 33, wherein the object is a camera.

43. The computer program product of claim 33, wherein combining the visual information and the sensor information includes synchronizing the visual information and the sensor information.

44. The computer program product of claim 33, wherein the virtual representation of the object is an online character of a video game.

45. The computer program product of claim 33, wherein the sensor information includes a time series of data provided by the sensor.

46. A system comprising:

a visual data collector for collecting visual information from a plurality of images, wherein each image includes content that represents a plurality of markers attached to a body suit;

a physical data collector for collecting sensor information provided by a plurality of inertial sensors also attached to the body suit; and

a computer system including,

a memory for storing the collected visual information and sensor information, and

a processor for executing instructions that perform a method comprising,

combining the visual information and the sensor information, and

determining, for each image, the respective positions of representations of the plurality of markers in a virtual representation of a pose provided by the body suit from the combined visual information and sensor information.

47. The system of claim 46, wherein the method further comprises:

tracking the movement of the plurality of markers across the plurality of images.

48. The system of claim 46, wherein determining the respective positions of the representations of the plurality of markers includes determining a region in at least one of the plurality of images that is likely to include a representation of the at least one of the plurality of markers based upon the sensor information provided by the plurality of inertial sensors.

49. The system of claim 46, wherein the method executed by the processor further comprises:

determining the respective positions of the representations of the plurality of markers includes determining initial positions for each of the plurality of markers based upon the collected sensor information.

50. The system of claim 46, wherein the visual data collector further comprises:

one or more cameras for collecting the plurality of images, wherein one or more inertial sensors are attached to each of the one or more cameras for collecting calibration information.

51. The system of claim 46, wherein the visual data collector further comprises:

one or more cameras for collecting the plurality of images, wherein the sensor information is used to calibrate the position of at least one of the cameras.

52. The system of claim 46, wherein the method executed by the processor further comprises:

producing a series of poses of the virtual representation of the body suit based upon the combined visual information and sensor information.