System and method for 3D projection mapping with robotically controlled objects

• US 9,832,352 B2
• Filed: 02/26/2014
• Issued: 11/28/2017
• Est. Priority Date: 11/16/2011
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: Autofuss, Co.
• Inventors: Linnell, Jeff
• Primary Examiner(s): Atala, Jamie
• Assistant Examiner(s): Abaza, Ayman
• Application Number:
  US14/190,091
• Publication Number:
  US 20140354787A1
• Time in Patent Office: 1,371 Days
• Field of Search: 345156, 348 51, 348 61
• CPC Class Codes: B25J 11/00 Manipulators not otherwise provided for
• CPC Class Codes: B25J 9/1633 compliant, force, torque control, e.g. combined with position control
• CPC Class Codes: G05B 2219/36429 Power assisted positioning
• CPC Class Codes: G05B 2219/39342 Adaptive impedance control
• CPC Class Codes: G05B 2219/39346 Workspace impedance control
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A system for motion control is presented. In one embodiment, a motion control 3D projection system includes a projector; and a projection surface coupled to a robotic arm, where the robotic arm moves the projection surface through a set of spatial coordinates, and a 3D projection from the projector is projected onto a set of coordinates of the projection surface and matches the 3D projection to the set of coordinates of the projection surface as the projection surface moves through the set of spatial coordinates. In additional embodiments, a master control system may integrate additional robotic arms and other devices to create a motion control scene with a master timeline.

Claims

1. A system comprising:

at least one projector located in an environment;

a projection surface located in the environment and coupled to a first device actor, wherein the first device actor is configured to move the projection surface through a set of spatial coordinates in the environment;

a camera located in the environment and coupled to a second device actor, wherein the camera is configured to capture image data of the environment or portions thereof as the projection surface moves through the set of spatial coordinates in the environment, and wherein the second device actor is configured to move the camera independently from the projector, through at least a portion of the set of spatial coordinates; and

a control system configured to;

(i) synchronize operation of the projector with operation of a shutter of the camera while (a) the projector projects a projection on the projection surface and (b) the camera captures image data of the environment including at least a portion of the projection surface with the projection thereon;

(ii) when the camera has a first pose at a first location in the set of spatial coordinates, cause the projector to render graphic content such that a three-dimensional effect of the projection of the graphic content on the projection surface corresponds to a first perspective of the projection surface from the first pose of the camera;

(iii) cause a movement of the camera to a second location in the set of spatial coordinates, which changes the position of the camera with respect to the projector;

(vi) determine a second pose of the camera at the second location;

(v) determine a second perspective of the projection surface from the camera, wherein the second perspective corresponds to the second pose of the camera at the second location; and

(vi) coordinate an update of the graphic content so as to maintain the three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the camera changes from the first perspective to the second perspective of the projection surface from the camera.

2. The system of claim 1, wherein the projector is coupled to a third device actor that is configured to move the projector, and wherein the control system is further configures to cause a movement of the projector such that the projection remains on the projection surface and as the projection surface moves through the set of spatial coordinates.

3. The system of claim 2, wherein the control system is configured to generate a control signal for the third device actor, wherein the control signal for the third device actor synchronizes movement of the projector with movement of the projection surface and movement of the camera.

4. The system of claim 1, wherein the control system is further configured to:

receive a signal indicating one or more shutter parameters corresponding to the operation of the shutter; and

generate a projector control signal based on the one or more shutter parameters.

5. The system of claim 1, wherein the control system is further configured to:

receive a signal indicating one or more projection parameters corresponding to the operation of the projector; and

generate, based on the one or more projection parameters, a shutter control signal for the camera.

6. The system of claim 1, wherein the control system is further configured to generate a shutter control signal to control the operation of the camera shutter, wherein the shutter control signal comprises one or more shutter parameters that substantially synchronize the operation of the shutter with the operation of the projector.

7. The system of claim 6, wherein the one or more shutter parameters comprise one or more of:

(a) a frame rate, (b) a phase offset, and (c) a trigger time.

8. The system of claim 1, wherein the control system is further configured to generate a projector control signal to control the operation of the projection, wherein the projector control signal comprises one or more projection parameter that substantially synchronize the operation of the shutter with the operation of the projector.

9. The system of claim 8, wherein the one or more projection parameters comprise one or more of:

(a) a frame rate, (b) a phase offset, and (c) a trigger time.

10. The system of claim 1, wherein the control system is further configured to:

determine both (a) one or more shutter parameters for the operation of the shutter and (b) one or more projection parameters for the operation of the projector, wherein the one or more shutter parameters and the one or more projection parameters are determined such that a rate at which the shutter operates is synchronized with a frame rate at which the projector operates;

generate a shutter control signal based on the one or more shutter parameters; and

generate a projector control signal based on one or more projection parameters for the operation of the shutter, wherein shutter control signal and the projector control signal are sent to the camera and the projector, respectively, to facilitate the camera and projector synchronizing the rate at which the shutter operates with the frame rate of the projector.

11. The system of claim 1, wherein the control system, the projector, or both is or are configured to keep the graphic content projected onto coordinates of the projection surface as the projection surface moves.

12. The system of claim 1, wherein the control system is further configured to render the graphic content such that the projection of the graphic content changes over time to maintain the three-dimensional effect of the projection of the graphic content on the projection surface, from the perspective of the projection surface from the camera, as the camera moves, as the projection surface moves, and as the camera and projection surface move concurrently.

13. The system of claim 12, wherein pose of the camera at a given time comprises coordinates of the camera, within the set of spatial coordinates, at the given time.

14. The system of claim 1, wherein the control system is further configured to:

cause a movement of the projection surface, wherein at least a portion of the movement of the projection surface is concurrent with at least a portion of the movement of the camera to the second location;

coordinate an update of the graphic content so as to maintain the three-dimensional effect of the projection of the graphic content on the projection surface, from the perspective of the projection surface from the camera, as perspective of the projection surface from the camera changes due to the combination of the movement of the camera and the movement of the projection surface.