Controlling a virtual camera

• US 8,964,052 B1
• Filed: 12/30/2010
• Issued: 02/24/2015
• Est. Priority Date: 07/19/2010
• Status: Active Grant

• Current Assignee: Lucasfilm Limited (The Walt Disney Company)
• Original Assignee: Lucasfilm Limited (The Walt Disney Company)
• Inventors: Wooley, Kevin, Sanders, Michael, Sullivan, Steve, Cantwell, Brian, Reynolds, Spencer
• Primary Examiner(s): Berhan, Ahmed A
• Application Number:
  US12/982,426
• Time in Patent Office: 1,517 Days
• Field of Search: 348/211.3, 348/211.99, 348/211.12, 348/211.13, 348/211.14, 348/159, 348/220.1, 348/207.99
• US Class Current: 348/211.99
• CPC Class Codes: G06F 3/0202 Constructional details or processes of manufacture of the input device
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0362 with detection of 1D translations or rotations of an operating part of the device, e.g. scroll wheels, sliders, knobs, rollers or belts
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G11B 27/00 Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Among other aspects, on computer-implemented method includes: receiving at least one command in a computer system from a handheld device; positioning a virtual camera and controlling a virtual scene according to the command; and in response to the command, generating an output to the handheld device for displaying a view of the virtual scene as controlled on a display of the handheld device, the view captured by the virtual camera as positioned.

Claims

1. A computer-implemented method comprising operating a computer system in one of a set of modes, wherein:

operating in a first mode comprises;

receiving in a computer system motion capture information from a handheld device and, in response, moving a virtual camera in a virtual scene and generating corresponding video output of the virtual scene, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene;

receiving in the computer system a command generated by a user activating a function on the handheld device;

ceasing movement of the virtual camera in the virtual scene in response to the command; and

subsequently resuming movement of the virtual camera in the virtual scene in response to motion capture input received from the handheld device; and

operating in a second mode comprises;

generating a second output to the handheld device that displays a view of the virtual scene on a display of the handheld device; and

modifying virtual assets defined for the virtual scene in response to inputs made on the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

2. A computer-implemented method comprising operating a computer system in one of a set of modes, wherein:

operating in a first mode comprises;

receiving in a computer system motion capture information from a handheld device and, in response, moving a virtual camera in a virtual scene and generating corresponding video output of the virtual scene, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene;

receiving in the computer system a command generated by a user activating a function on the handheld device;

repositioning the virtual camera in the virtual scene in response to the command independent of the motion capture information; and

subsequently controlling the virtual camera according to motion capture input received from the handheld device; and

operating in a second mode comprises;

generating a second output to the handheld device that displays a view of the virtual scene on a display of the handheld device; and

modifying virtual assets defined for the virtual scene in response to inputs made on the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

3. A computer-implemented method comprising operating a computer system in one of a set of modes, wherein:

operating in a first mode comprises;

receiving in a computer system motion capture information from a handheld device and, in response, moving a virtual camera in a virtual scene and generating corresponding video output of the virtual scene, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene;

receiving in the computer system a command generated by a user activating a function on the handheld device;

in response to the command, constraining the virtual camera in the virtual scene relative to an asset in the virtual scene such that the virtual camera undergoes the same or similar movement as the asset in the virtual scene; and

generating video output with the constrained virtual camera; and

operating in a second mode comprises;

generating a second output to the handheld device that displays a view of the virtual scene on a display of the handheld device; and

modifying virtual assets defined for the virtual scene in response to inputs made on the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

4. A computer-implemented method comprising:

receiving at least one command in a computer system from a handheld device, the command causing the computer system to operate in at least one of a set of modes;

wherein;

operating in a first mode comprises;

positioning a virtual camera in a virtual scene, controlling the virtual scene according to motion capture information received from the handheld device, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene; and

in response to the motion capture information, generating an output to the handheld device for displaying a view of the virtual scene as controlled on a display of the handheld device, the view captured by the virtual camera as positioned; and

operating in a second mode comprises;

generating a second output to the handheld device that displays at least a view of the virtual scene on a display of the handheld device; and

modifying virtual assets defined for the virtual scene in response to inputs made on the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

5. The computer-implemented method of claim 4, wherein operating in the first mode further comprises:

recording a snapshot of the virtual scene using the virtual camera in response to the command.

6. The computer-implemented method of claim 5, further comprising recording a storyboard, an animated camera path or a state of the virtual scene using the snapshot.

7. The computer-implemented method of claim 4, wherein the command is generated using a touchscreen on the handheld device.

8. The computer-implemented method of claim 4, further comprising selecting another virtual camera in the virtual scene based on a user input made on the handheld device and generating another view of the virtual scene with the other virtual camera.

9. The computer-implemented method of claim 4, wherein the output is generated substantially in real time with the command.

10. The computer-implemented method of claim 9, wherein operating in either the first or the second mode further comprises generating a 3D placeholder point in the virtual scene based on a user input made on the handheld device.

11. The computer-implemented method of claim 4, wherein the display includes a touchscreen, and wherein at least the second mode further comprises receiving and storing a digital overlay record for the view generated using the touchscreen.

12. The computer-implemented method of claim 4, wherein at least the second mode further comprises generating an animation including the view, and editing the animation based on a user input made on the handheld device.

13. The computer-implemented method of claim 12, wherein the editing comprises setting a start frame or an end frame for the animation.

14. The computer-implemented method of claim 12, further comprising applying handheld motion of the handheld device to the animation using the motion capture input.

15. The computer-implemented method of claim 4 wherein operating in either the first or the second mode further comprises recording a camera angle in the computer system using the motion capture input.

16. The computer-implemented method of claim 4, wherein the motion capture input is generated by the handheld device in response to a user performing a gesture, and further comprising performing at least one action on the virtual scene or the virtual camera based on the motion capture input.

17. The computer-implemented method of claim 4, further comprising receiving audio recorded using the handheld device, and storing the audio in association with the virtual scene.

18. The computer-implemented method of claim 4, further comprising receiving video of a physical scene recorded using the handheld device.

19. The computer-implemented method of claim 4, wherein operating in the first mode comprises:

operating the handheld device in a motion capture stage, wherein the virtual scene includes a representation of the motion capture stage, and

aligning the representation with at least one physical object detected in the motion capture stage.

20. The computer-implemented method of claim 19 wherein the detected physical object is aligned with a virtual object in the representation.

21. The computer-implemented method of claim 20, wherein the virtual object is another virtual camera, a character or a set piece.

22. The computer-implemented method of claim 19, wherein multiple physical objects are detected on the motion capture stage and aligning the representation further comprises performing a best-fit algorithm that seeks a position for the virtual camera that differs least from a physical relationship between the handheld device and the detected multiple physical objects.

23. A computer program product tangibly embodied in a non-transitory computer-readable storage medium and comprising instructions that when executed by a processor perform a method comprising:

receiving at least one command in a computer system from a handheld device, the command causing the computer system to operate in at least one of a set of modes;

wherein

operating in a first mode comprises;

positioning a virtual camera in a virtual scene, controlling the virtual scene according to motion capture information received from the handheld device, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene; and

in response to the motion capture information, generating an output to the handheld device for displaying a view of the virtual scene as controlled on a display of the handheld device, the view captured by the virtual camera as positioned; and

operating in a second mode comprises;

generating an output to the handheld device for displaying a view of the virtual scene on a display of the handheld device; and

modifying virtual assets defined for the virtual scene in response to inputs made on the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

24. A system comprising:

a computer subsystem; and

a handheld device communicatively linked with the computer subsystem and comprising a display configured to display images and a camera configured to record images,

wherein the computer subsystem comprises code that causes the computer subsystem to perform a method comprising;

receiving a first command from the handheld device, wherein the first command causes the computer subsystem to;

position a virtual camera in a virtual scene and control the virtual scene according to motion information of the handheld device, wherein the virtual camera is an abstract definition including a point in space and a direction for a field of view within the virtual scene; and

in response to the motion information, generate an output to the handheld device for displaying a view of the virtual scene as controlled on the display of the handheld device, the view captured by the virtual camera as positioned; and

receiving a second command from the handheld device, wherein the second command causes the computer subsystem to modify virtual assets defined for the virtual scene in response to inputs received from the handheld device, wherein modifying includes adding, relocating, removing or altering an appearance of at least one of the virtual assets defined for the virtual scene.

25. The system of claim 24 wherein the method further comprises generating corresponding video output of the virtual scene in response to the motion information and displaying the video output on the display of the handheld device.

26. The system of claim 25 wherein the method further comprises recording the generated video output.

27. The system of claim 24, wherein the display comprises a touchscreen.