System for filming a video movie

• US 9,756,277 B2
• Filed: 07/02/2015
• Issued: 09/05/2017
• Est. Priority Date: 12/13/2011
• Status: Active Grant

• Current Assignee: Oaro Co.
• Original Assignee: SolidAnim
• Inventors: Partouche, Isaac, Szlapka, Jean-Franois, Linot, Emmanuel
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Wendmagegn, Girumsew
• Application Number:
  US14/790,099
• Publication Number:
  US 20150358508A1
• Time in Patent Office: 796 Days
• Field of Search: 386227
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

System for filming a video movie in a real space includes a filming camera, a sensor, a computerized pinpointing module for determining the location of the filming camera, a monitoring screen and a computerized compositing module for generating on the monitoring screen a composite image of the real image and of a projection of a virtual image, generated according to the filming camera location data.

Claims

1. System for filming video footage in a real space defined in a real reference system, wherein the system for filming video footage comprises:

a filming camera, dedicated to the task of filming, movable in a guided manner in the real space or being movable about within the real space by being carried by an operator, suitable for recording a real image for a plurality of discrete time frames,

a location pinpointing system, comprising;

at least one sensor, dedicated to the task of pinpointing, comprising an optical camera having an acquisition frequency that is greater than 100 images per second, using a wide angle lens that has a capture angle exceeding 160 degrees, and with a field of view overlapping a field of view of the filming camera, wherein the optical camera is provided at a location relative to the filming camera that is known for each time frame via location data, and suitable for transmitting to a computerized pinpointing module the natural topographical information detected by the sensor,

the computerized pinpointing module suitable for determining, for each time frame, the filming camera location data in the real reference system based on the location data of the optical camera, and on a comparison of the natural topographical information and a predetermined three-dimensional model of the real space,

a monitoring screen,

a computerized compositing module suitable for generating on the monitoring screen, for each time frame, a composite image of the real image and of a projection of a virtual image taken from a database of virtual animations, said projection generated according to the filming camera location data in the real reference system, and

wherein, if the field of view of the sensor has been obstructed by a real element in the real space, the computerized pinpointing module determines the position of the filming camera in the real space at any time.

2. System for filming video footage according to claim 1, wherein the pinpointing system comprises a computerized generation module suitable for generating said predetermined three-dimensional model of the real space, and wherein said sensor is suitable for transmitting to the computerized generation module topographical information detected by the sensor.

3. System for filming video footage according to claim 2, wherein the sensor is suitable for simultaneously transmitting to the computerized pinpointing module and to the computerized generation module natural topographical information detected by the sensor, and wherein the computerized generation module is suitable for enriching said predetermined three-dimensional model of the real space using natural topographical information detected by the sensor.

4. System for filming video footage according to claim 1, wherein the topographical information comprises information relating to geometric objects of the real space that are chosen from among points, lines, surfaces, and volumes.

5. System for filming video footage according to claim 1, wherein, in the filming configuration, the filming camera and the sensor are fixedly attached to each other.

6. System for filming video footage according to claim 1, further comprising a positioning system comprising a positioning pattern suitable for simultaneous detection by the filming camera and by the sensor in a positioning configuration, and a computerized positioning module suitable for determining the respective location data of the sensor and of the filming camera from their simultaneous detection of the positioning pattern.

7. System for filming video footage according to claim 1, further comprising an optical calibration system comprising an optical calibration pattern suitable for detection by the filming camera, in an optical calibration configuration, and wherein the computerized pinpointing module is suitable for determining, for each time frame, the filming camera location data in the real reference system based in addition on optical calibration data of the filming camera that are determined by the optical calibration system.

8. System for filming video footage according to claim 1, further comprising an inertial sensor fixed to the filming camera, suitable for determining a movement of the filming camera, the computerized pinpointing module being suitable for determining the filming camera location data in the real reference system based in addition from data supplied by the inertial sensor.

9. System for filming video footage according to claim 1, further comprising a computerized animation module comprising a database of virtual animations, each animation comprising, for each of a set of time frames, a three-dimensional image expressed in a virtual reference system, the computerized animation module being suitable for transmitting said three-dimensional images to the compositing module.

10. System for filming video footage according to claim 9, wherein the computerized pinpointing module is suitable for transmitting the predetermined three-dimensional model of the real space to the computerized animation module.

11. System for filming video footage according to claim 9, wherein the computerized compositing module is suitable for generating a shadow of the virtual image on the monitoring screen, for each time frame, said shadow being generated according to the filming camera location data in the real reference system and lighting location data in the real reference system.

12. System for filming video footage according to claim 1, wherein, for a later time frame, the computerized pinpointing module is able to determine the filming camera location data in the real reference system based in addition on the filming camera location data in the real reference system for an earlier time frame.

13. System for filming video footage according to claim 1, wherein the computerized pinpointing module comprises a selection module suitable for selecting, from among the geometric patterns, geometric patterns of the three-dimensional model that are usable for locating the position of the filming camera in 3D space.

14. System for filming video footage according to claim 13, wherein the selection module compares geometric patterns of a later image with geometric patterns of an earlier image, associates geometric patterns present in both images and unmoving in the real space, and does not keep the other geometric patterns for comparison with the three-dimensional model.

15. System for filming video footage according to claim 1, wherein the pinpointing system further comprises a second sensor having at least one characteristic that is different from the first sensor, chosen from among the position, the orientation, the solid angle of view, the acquisition frequency, the optical axis, and the field of view.

16. System for filming video footage according to claim 1, wherein the sensor of the pinpointing system has an acquisition frequency that is greater than an acquisition frequency of the filming camera.

17. System for filming video footage according to claim 1, wherein the sensor of the pinpointing system has an acquisition in black and white.

18. System for filming video footage according to claim 1, wherein the sensor of the pinpointing system has a bulk at least two times smaller than the bulk of the filming camera.

19. System for filming video footage according to claim 1, wherein the sensor of the pinpointing system has an optical axis parallel to an optical axis of the filming camera.

20. System for filming video footage according to claim 1, further comprising a measurement reference, filmable by the filming camera, the computerized compositing module being able to bring a virtual image to the scale of the real space based on an image of the measurement reference captured by the filming camera.

21. System for filming video footage according to claim 1, further comprising a system for determining a magnification setting for the filming camera, the computerized compositing module being suitable for generating the composite image by taking into account said parameter.

22. System for filming video footage according to claim 1, wherein the real space may as well be an outdoor space as an inside space.

23. System for filming video footage according to claim 1, wherein the real space is an outdoor space.

24. System for filming video footage according to claim 1, wherein the real space is an inside space.

25. System for filming video footage according to claim 1, wherein, in addition to points, the three-dimensional model also comprises a set of lines/surfaces/volumes, which are determined either by the generation module itself or by assistance from a generation module operator, said generation module operator indicating to the generation module that a set of geometric objects are part of the same line/surface/volume.

26. System for filming video footage according to claim 1, wherein the three-dimensional model is used to manage objects in the real space and virtual objects covering each other.

27. System for filming video footage according to claim 1, wherein the filming camera is movable in a guided manner in the real space.

28. System for filming video footage according to claim 27, wherein the filming camera is movable in a guided manner in the real space by being mounted on a rail or a crane with a hinged arm defining the possible positions for the filming camera.

29. System for filming video footage according to claim 1, wherein the filming camera is movable about within the real space by being carried by an operator.

30. System for filming video footage in a real space defined in a real reference system, wherein the system for filming video footage comprises:

a filming camera, dedicated to the task of filming, movable in a guided manner in the real space or being movable about within the real space by being carried by an operator, suitable for recording a real image for a plurality of discrete time frames,

a location pinpointing system, comprising;

at least one sensor, dedicated to the task of pinpointing, comprising an optical camera having an acquisition frequency that is greater than 100 images per second, using a wide angle lens that has a capture angle exceeding 160 degrees, and with a field of view overlapping a field of view of the filming camera, wherein the optical camera is provided at a location relative to the filming camera that is known for each time frame via location data, and suitable for transmitting to a computerized pinpointing module the natural topographical information detected by the sensor,

the computerized pinpointing module suitable for determining, for each time frame, the filming computerized pinpointing module suitable for determining, for each time frame, the filming camera location data in the real reference system based on the location data of the optical camera, and on a comparison of the natural topographical information and a predetermined three-dimensional model of the real space,

a monitoring screen,

a computerized compositing module suitable for generating on the monitoring screen, for each time frame, a composite image of the real image and of a projection of a virtual image taken from a database of virtual animations, said projection generated according to the filming camera location data in the real reference system, and

wherein the pinpointing system comprises a computerized generation module suitable for generating said predetermined three-dimensional model of the real space, and wherein said sensor is suitable for transmitting to the computerized generation module topographical information detected by the sensor, and for simultaneously transmitting to the computerized pinpointing module and to the computerized generation module natural topographical information detected by the sensor, and wherein the computerized generation module is suitable for enriching said predetermined three-dimensional model of the real space using natural topographical information detected by the sensor.

31. System for filming video footage in a real space defined in a real reference system, wherein the system for filming video footage comprises:

a filming camera, dedicated to the task of filming, movable in a guided manner in the real space or being movable about within the real space by being carried by an operator, suitable for recording a real image for a plurality of discrete time frames,

an optical calibration system comprising an optical calibration pattern suitable for detection by the filming camera, in an optical calibration configuration,

a location pinpointing system, comprising;

at least one sensor, dedicated to the task of pinpointing, comprising an optical camera having an acquisition frequency that is greater than 100 images per second, using a wide angle lens that has a capture angle exceeding 160 degrees, and with a field of view overlapping a field of view of the filming camera, wherein the optical camera is provided at a location relative to the filming camera that is known for each time frame via location data, and suitable for transmitting to a computerized pinpointing module the natural topographical information detected by the sensor,

the computerized pinpointing module suitable for determining, for each time frame, the filming camera location data in the real reference system based on the location data of the optical camera, on optical calibration data of the filming camera that are determined by the optical calibration system, and on a comparison of the natural topographical information and a predetermined three-dimensional model of the real space,

a monitoring screen,

a computerized compositing module suitable for generating on the monitoring screen, for each time frame, a composite image of the real image and of a projection of a virtual image taken from a database of virtual animations, said projection generated according to the filming camera location data in the real reference system.

32. System for filming video footage in a real space defined in a real reference system, wherein the system for filming video footage comprises:

a filming camera, dedicated to the task of filming, movable in a guided manner in the real space or being movable about within the real space by being carried by an operator, suitable for recording a real image for a plurality of discrete time frames,

a location pinpointing system, comprising;

at least one sensor, dedicated to the task of pinpointing, comprising an optical camera having an acquisition frequency that is greater than 100 images per second, using a wide angle lens that has a capture angle exceeding 160 degrees, and with a field of view overlapping a field of view of the filming camera, wherein the optical camera is provided at a location relative to the filming camera that is known for each time frame via location data, and suitable for transmitting to a computerized pinpointing module the natural topographical information detected by the sensor,

the computerized pinpointing module suitable for determining, for each time frame, the filming camera location data in the real reference system based on the location data of the optical camera, and on a comparison of the natural topographical information and a predetermined three-dimensional model of the real space,

a monitoring screen,

a computerized compositing module suitable for generating on the monitoring screen, for each time frame, a composite image of the real image and of a projection of a virtual image taken from a database of virtual animations, said projection generated according to the filming camera location data in the real reference system, and

wherein, in addition to points, the three-dimensional model also comprises a set of lines/surfaces/volumes, which are determined either by the generation module itself or by assistance from a generation module operator, said generation module operator indicating to the generation module that a set of geometric objects are part of the same line/surface/volume.