Method and system for virtual cinematography

• US 6,040,841 A
• Filed: 08/02/1996
• Issued: 03/21/2000
• Est. Priority Date: 08/02/1996
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Li-wei, Cohen, Michael F., Salesin, David H.
• Primary Examiner(s): Feild, Joseph H.
• Application Number:
  US08/691,996
• Time in Patent Office: 1,327 Days
• Field of Search: 345/427, 345/355, 345/419, 345/302, 345/473-475, 345/328
• US Class Current: 345/473
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6669 using a plurality of virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character change rooms
• CPC Class Codes: A63F 2300/6684 by dynamically adapting its position to keep a game object in its viewing frustrum, e.g. for tracking a character or a ball
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

Communication in a three-dimensional virtual environment generated by a computer is achieved by automatically applying rules of cinematography typically used for motion pictures. The cinematographic rules are codified as a hierarchical finite state machine, which is executed in real-time by a computer in response to input stimulation from a user or other source. The finite state machine controls camera placements automatically for a virtual environment. The finite state machine also exerts subtle influences on the positions and actions of virtual actors, in the same way that a director might stage real actors to compose a better shot. An implementation of a real-time camera controller based on the finite state machine for automatic virtual cinematography, called the virtual cinematographic application module (VC) is presented. The VC is used in virtual reality and other interactive applications to improve upon the fixed point-of-view shots or ceiling mounted cameras that such applications typically employ today. VC also helps improve the next generation of "intelligent-agent" user interfaces, by allowing the users to see themselves with an agent at camera positions that appear natural.

Claims

1. A method for displaying virtual cinematography on a display, the method comprising:

receiving cinematographic events in real time using a real-time application module;

interpreting the received cinematographic events in real time using ciniematographic logic; and

automatically producing a camera view specification from the interpreted cinematographic events, where the camera specification is used to view a virtual scene on the display.

2. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 1.

3. The method of claim 1 where the camera view specification includes at least one specification chosen from a set comprising apex, closeapex, apex3, external, 2long, external1to2, exclose2, internal, full, tracking, panning, follow, fixed, raised, group, subjective, subjectzoom, pov, and null camera view specifications.

4. The method of claim 1 where the camera view specification includes selecting camera view distances.

5. The method of claim 4 where the camera view distances include at least one distance chosen from a set comprising an extreme closeup view distance, a view distance, a medium view distance, a full view distance, and a long view distance.

6. The method of claim 1 where the cinematographic logic includes film heuristics for selecting film shots.

7. The method of claim 6 where the film heuristics include at least one heuristic chosen from a set comprising Don'"'"'t cross the Line, Avoid jump cuts, Use establishing shots, Let the actor lead, Break movement, and Maintain continuous movement.

8. The method of claim 1 where the cinematographic logic includes camera modules and cinematographic idioms implemented as a hierarchical state machine that determines when to switch camera views.

9. The method of claim 1 further comprising adjusting a cinematographic character within a cinematographic event to accord it with the camera view specification.

10. A computer system for implementing virtual cinematography, the system comprising:

a real-time application module for producing cinematographic events in response to land interfacing with al user input which is received in real time on a computer;

a virtual cinematographic module interpreting the cinematographic events in real time and automatically determining camera position based on the cinematographic events; and

a renderer application module for rendering a virtual environment in response to commands from the virtual cinematographic module.

11. The system of claim 10 where the virtual cinematographic module includes camera modules and virtual cinematographic idioms implemented as a hierarchical state machine that decides when to switch camera views.

12. A method for displaying a virtual actor to a user on a computer using virtual cinematography, the method comprising:

displaying a virtual actor on a computer;

automatically selecting a virtual camera view specification;

automatically selecting a virtual camera view distance; and

modifying the arrangement and actions of the virtual actor in real time to conform to the selected virtual camera view specification and virtual camera view distance.

13. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 12.

14. The method of claim 12 where the virtual camera specifications include at least one specification chosen from a set comprising apex, closeapex, apex3, external, 2long, external1to2, exclose2, internal, full, tracking, panning, follow, fixed, raised, group, subjective, subjectzoom, pov, and null camera view specifications.

15. The method of claim 12 where the camera view distances include at least one distance chosen from a set comprising extreme closeup view distance, a closeup view distance, a medium view distance, a full view distance, and a long view distance.

16. A method of creating an individual virtual cinematographic idiom for use in a computer system, the method comprising:

defining one or more individual virtual cinematographic idiom states;

selecting a camera module to associate with each defined individual virtual cinematographic idiom state; and

defining idiom actions which include transitions between the defined individual virtual cinematographic idiom states;

wherein the idiom actions are used to display a virtual scene in real time on a computer in the computer system.

17. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 16.

18. The method of claim 16 where the camera modules include at least one view chosen from a set comprising apex, closeapex, apex3, external, 2long, external1to2, exclose2, internal, full, tracking, panning, follow, fixed, raised, group, subjective, subjectzoom, pov, and null camera module views.

19. The method of claim 16 where the individual virtual cinematographic idiom actions includes the registration and raising of exceptions.

20. A method of creating a complex virtual cinematographic idiom for use in a computer system to display a virtual scene, the method comprising:

defining one or more complex virtual cinematographic idiom states;

selecting an individual virtual cinematographic idiom for one or more of the defined complex virtual cinematographic idiom states where the individual idiom includes one or more individual idiom states, one or more camera modules and individual idiom actions;

selecting a camera module to associate with each defined virtual cinematographic idiom state;

defining complex idiom actions which include transitions between the defined complex virtual cinematographic idiom states; and

defining conditions which are tested to decide which transition occurs;

wherein in the complex virtual cinematographic idiom is used to display a virtual scene in real time on a computer.

21. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 20.

22. A method of creating high level virtual cinematographic idioms for use in a computer system to display a virtual scene, the method comprising:

selecting a first set of individual virtual cinematographic idioms, where each of the individual virtual cinematographic idioms includes individual virtual cinematographic idiom states, camera modules, and virtual cinematographic individual idiom actions;

combining the first set of individual virtual cinematographic idioms to create a complex virtual cinematographic idiom, where the complex virtual cinematographic idiom includes complex virtual cinematographic idiom states, camera modules, and complex virtual cinematographic idiom actions;

selecting a second set of individual virtual cinematographic idioms and the complex virtual cinematographic idiom; and

combining the second set of individual virtual cinematographic idioms and the complex virtual cinematographic idiom to create a high level virtual cinematographic idiom;

wherein the high-level virtual cinematographic idioms are used to display a virtual scene in a real-time computer.

23. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 22.

24. The method of claim 22 where the high level virtual cinematographic idiom is a verb virtual cinematographic idiom.

25. A computer-readable storage medium having stored thereon instructions causing a computer to perform a method comprising:

selecting a set of verb virtual cinematographic idioms by identifying at least a verb and an object for each verb virtual cinematographic idiom;

combining the selected set of verb virtual cinematographic idioms to create a pluralities of virtual movie scenes; and

combining selected ones of the virtual movie scenes to create a virtual movie.

26. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 25.

27. A method of virtual directing for display of a virtual scene on a computer display, the method comprising:

selecting a set of virtual cinematographic idioms, an idiom being a hierarchical state machine that automatically determines how camera viewing directions transition from one shot to another, where the virtual cinematographic idioms include one or more virtual cinematographic idiom states, one or more camera modules and individual idiom actions;

changing the camera modules within the virtual cinematographic idioms based on a specific film directing style to create a customized, modified set of virtual cinematographic idioms;

combining the modified set virtual cinematographic idioms based on the specific film directing style to create a pluralities of virtual movie scenes; and

combining selected ones of the virtual movie scenes to create a virtual movie for display on the computer display.

28. A computer-readable storage medium having stored therein instructions capable of causing a computer to perform the method of claim 27.

29. A system for displaying a virtual scene on a client computer, comprising:

a real-time application module that receives user input from a user on the client computer to control, in real-time, a virtual actor displayed on the client computer as seen from a virtual camera;

a real-time camera controller that includes a hierarchical finite state machine and a plurality of camera modules, the state machine for determining the positioning of the virtual camera and for determining display information for the virtual actor by invoking one of the camera modules; and

a renderer for rendering the virtual actor using camera position information and display information from the real-time controller.

30. The system of claim 29 wherein a user can control the virtual actor through a user input device.

31. The system of claim 30 further including a plurality of client computers connected to a network, wherein the first mentioned client computer is connected to the network and wherein at least two client computers each have a virtual actor in the virtual scene wherein each virtual actor is separately controlled by its associated client computer.