Apparatus and method for providing content experience service

• US 9,236,032 B2
• Filed: 07/31/2014
• Issued: 01/12/2016
• Est. Priority Date: 09/13/2013
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Koo, Bonki, Choi, Yoonseok, Jung, Soonchul, Choi, Jinsung
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US14/449,036
• Publication Number:
  US 20150078621A1
• Time in Patent Office: 530 Days
• Field of Search: 382/103, 382/107, 345/633
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 21/2187 Live feed

Abstract

A method and apparatus for providing content experience service are disclosed. The apparatus includes a camera device tracking unit, a user behavior tracking unit, a real image acquisition unit, a motion information processing unit, a virtual space control unit, and a virtual multi-image generation unit. The camera device tracking unit collects camera motion information. The user behavior tracking unit collects user motion information. The real image acquisition unit photographs a space including the user, and separates a real image into a foreground and a background using a background key table. The motion information processing unit corrects the camera motion information and the user motion information. The virtual space control unit generates virtual space control information and virtual space information. The virtual multi-image generation unit generates a virtual multi-image, and provides the content experience service based on the generated virtual multi-image.

Claims

1. A method of providing content experience service, comprising:

collecting camera motion information, corresponding to results obtained by tracking positions and directions of an immersive display camera device and an inspection view camera device placed in a studio space, and user motion information corresponding to a position and direction of a user;

photographing a space including the user, and separating a real image corresponding to photographed results into a foreground and a background using a background key table;

correcting the camera motion information and the user motion information;

generating virtual space control information and virtual space information based on the corrected camera motion information and user motion information; and

generating a virtual multi-image based on the virtual space control information and the virtual space information, and providing the content experience service based on the generated virtual multi-image.

2. The method of claim 1, wherein providing the content experience service comprises:

generating a first person viewpoint immersion image provided to the immersive display camera device worn by the user; and

generating a spectator viewpoint image output to an inspection view image device corresponding to spectators other than the user.

3. The method of claim 1, wherein collecting the camera motion information and the user motion information comprises collecting the camera motion information and the user motion information using a marker-based tracking system within the studio space.

4. The method of claim 1, wherein separating the real image into the foreground and the background comprises:

capturing the real image of the user in order to provide spectators with an immersion environment of the user;

setting characteristics of the real image to be separated as a background in order to separate the real image into a foreground corresponding to an area of interest to be used in images of the content experience service and the background not corresponding to the area of interest; and

if the inspection view camera device moves while capturing images, comparing a position and direction of the inspection view camera device with a position and direction stored in a background key table and then determining a final background key value.

5. The method of claim 1, wherein generating the virtual space control information and the virtual space information comprises:

generating the virtual space control information that forms a virtual space based on the camera motion information and the user motion information;

controlling attributes of a camera that functions as a camera in the virtual space;

controlling a motion of an avatar corresponding to a three-dimensional (3D) model that substitutes for a shape of the user within the virtual space; and

recognizing a specific gesture based on the user motion information, and controlling objects that form the virtual space based on results of the recognition.

6. An apparatus for providing content experience service, comprising:

a camera device tracking unit configured to collect camera motion information corresponding to results obtained by tracking positions and directions of an immersive display camera device and an inspection view camera device placed in a studio space;

a user behavior tracking unit configured to collect user motion information corresponding to a position and direction of a user;

a real image acquisition unit configured to photograph a space including the user, and to separate a real image corresponding to photographed results into a foreground and a background using a background key table;

a motion information processing unit configured to correct the camera motion information and the user motion information;

a virtual space control unit configured to generate virtual space control information and virtual space information based on the corrected camera motion information and user motion information; and

a virtual multi-image generation unit configured to generate a virtual multi-image based on the virtual space control information and the virtual space information, and to provide the content experience service based on the generated virtual multi-image.

7. The apparatus of claim 6, wherein the camera device tracking unit comprises:

a camera device motion tracking unit configured to track the positions and directions of the immersive display camera device and the inspection view camera device using a marker-based tracking system within the studio space;

a camera device motion information transmission unit configured to transfer the camera motion information, corresponding to the results obtained by tracking the camera device motion tracking unit, to an image server over a network; and

a camera device motion information management unit configured to manage the camera motion information according to each identifier of a camera.

8. The apparatus of claim 6, wherein the user behavior tracking unit comprises:

a user motion collection unit configured to collect the user motion information;

a user motion transmission unit configured to transfer the user motion information to an image server over a network; and

a user motion management unit configured to manage the user motion information based on a role of each of the markers attached on the user.

9. The apparatus of claim 6, wherein the real image acquisition unit comprises:

a real image photographing unit configured to capture the real image using an image device;

an image background setting unit configured to set characteristics of the real image in order to separate the real image into the foreground and the background; and

a foreground image management unit configured to, if the inspection view camera device moves while capturing images, compare a position and direction of the inspection view camera device with a position and direction stored in a background key table and then determine a final background key value.

10. The apparatus of claim 6, wherein the virtual space control unit comprises:

a virtual space control data generation unit configured to generate the virtual space control information that forms a virtual space based on the camera motion information and the user motion information;

a camera attribute control unit configured to control attributes of a camera that functions as a camera in the virtual space;

an avatar motion control unit configured to control a motion of an avatar corresponding to a 3D model that substitutes for a shape of the user within the virtual space;

a user gesture recognition unit configured to recognize a specific gesture based on the user motion information; and

a virtual space interaction control unit configured to control objects that form the virtual space based on results recognized by the user gesture recognition unit.

11. The apparatus of claim 6, wherein the virtual multi-image generation unit comprises a spectator view generation unit configured to, if the user is designated as an area of interest in the real image, control a depth position of the real image based on settings when an image of the user overlaps an image of a virtual space.