System and method for visualizing synthetic objects within real-world video clip

• US 9,586,141 B2
• Filed: 09/10/2012
• Issued: 03/07/2017
• Est. Priority Date: 09/08/2011
• Status: Active Grant

• Current Assignee: BowFlex, Inc.
• Original Assignee: Paofit Holdings Pte. Ltd. (BowFlex, Inc.)
• Inventors: Hardy, Marc Scott, Frank, Andrei Richard
• Primary Examiner(s): Suhol, Dmitry
• Assistant Examiner(s): LARSEN, CARL VICTOR
• Application Number:
  US14/343,378
• Publication Number:
  US 20140228118A1
• Time in Patent Office: 1,639 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/245 specially adapted to a particular type of game, e.g. steering wheels
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/61 using advertising information
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/816 Athletics, e.g. track-and-field sports
• CPC Class Codes: A63F 2300/203 Image generating hardware
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6009 for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6653 for altering the visibility of an object, e.g. preventing the occlusion of an object, partially hiding an object
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8005 Athletics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Embodiments of the present invention provide a system and a method for visualizing synthetic objects within a real-world video clip. The embodiments allow interactive, synthetic objects or characters to appear as if they are “embedded” inside or form part of a video clip. These objects move independently of the environment within the video clip and are controlled by a user. In one embodiment the video frames are set as textures in the furthermost layer in a 3D environment viewer or 3D “engine”. In another embodiment separate layers of video frames are superimposed with the synthetic objects on an output screen and synchronizing the movements and the field of view settings to give the perception that only one camera is used.

Claims

1. A method of visualizing synthetic objects within a real-world video clip, comprising:

a. building into a transparent 3D scene, via a processor, at least one of one or more objects and features from the real-world video clip, wherein the objects and features are at least one of static and dynamic based on the real-world video clip;

b. visualizing, in a viewer application, the built transparent 3D scene and dynamic objects overlaid on a background layer of the real-world video clip;

c. adding additional synthetic objects, static or dynamic, within a transparent 3D scene representation of the features in the real-world video clip, the transparent 3D scene representation being based on the visualization of the built transparent 3D scene;

d. using a 3D transparent scene as an occlusion layer, wherein an object or part of the occlusion layer appearing behind a transparent object is occluded and occupying a place with the real-world video clip; and

e. controlling the movement of the synthetic objects independent from the movement of the video clip, wherein the viewer application adjusts a speed of changing of individual frames of the video clip from zero FPS (frames per second) up to rates of 60 FPS to simulate a standard video player and to achieve a high rate of control over the speed of movement through the video clip.

2. The method of claim 1 further comprising:

extracting information from at least a portion of the real-world video clip, the extracted information being processed to form a transparent 3D scene reflecting a route travelled by a camera in the real-world video clip.

3. The method of claim 2, wherein the extracted information comprises:

a. a 3D camera movement coordinates and a path from the real-world video clip or sections of the real-world video clip; and

b. a 3D spatial or “

point cloud”

coordinate data of objects from the real-world video clip.

4. The method of claim 3, wherein the extracted information is processed by merging the 3D camera movement coordinates and path with the 3D spatial coordinate data to form the transparent 3D scene.

5. The method of claim 2 further comprising applying the information extracted from the real-world video clip to a camera of the viewer application.

6. The method of claim 1 further comprising enabling a user to select a path from the real world video clip or from one or more other real-world video clips for visualizing the synthetic objects.

7. The method of claim 1 further comprising building into the transparent 3D scene additional static features and refining existing features from the real-world video clip.

8. The method of claim 1 further comprising visualizing the real-world video clip within the viewer application as the furthermost layer in a 3D space.

9. The method of claim 8, wherein individual frames of the video clip are rendered as textures on an object in the furthermost layer in the 3D space, wherein the textures are changeable at adjustable rates to simulate a standard video player.

10. The method of claim 1 wherein a result to the viewer application is the visual appearance that the synthetic objects appear to exist or move within the landscape depicted in the video.

11. The method of claim 1 further comprising enabling at least one of a user and one or more controlling devices to control independently the synthetic objects.

12. The method of claim 1, wherein the viewer application allows rendering of the video clip and 3D objects on a screen in separate layers, each of the separate layers synched in a 3D space.

13. The method of claim 1 further comprising displaying an HTML browser layer on a screen in a 3D space.

14. The method of claim 1 further comprising allowing placement of objects, containing at least one of advertising images, objects or video clips, in a way to give an appearance of existence thereof in an environment depicted in the video clip.

15. The method of claim 1 wherein the viewer application includes functions allowing objects resembling doorways or portals to be placed in a way to give an appearance of existence thereof in the environment depicted in the video.

16. The method of claim 1 further comprising providing one or more functionalities comprising point systems, characters, and sound effects for usage over the real-world video clip.

17. The method of claim 1 further comprising allowing a user to change a background video when the user chooses to change a direction of a route in the real-world video clip.

18. A computer system configured to perform the method recited in claim 1.