Method and system for enhancing virtual stage experience

• US 7,053,915 B1
• Filed: 07/16/2003
• Issued: 05/30/2006
• Est. Priority Date: 07/30/2002
• Status: Active Grant

• Current Assignee: S Aqua Semiconductor LLC (Intellectual Ventures LLC)
• Original Assignee: Advanced Interfaces Incorporated
• Inventors: Jung, Namsoon, Sharma, Rajeev
• Primary Examiner(s): Luu, Matthew
• Application Number:
  US10/621,181
• Time in Patent Office: 1,049 Days
• Field of Search: 345629-641, 345/473, 345/474, 434/307.A, 434/308, 434/309, 434/314, 382/284
• US Class Current: 345/633
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part

Abstract

The present invention is a system and method for increasing the value of the audio-visual entertainment systems, such as karaoke, by simulating a virtual stage environment and enhancing the user'"'"'s facial image in a continuous video input, automatically, dynamically and in real-time. The present invention is named Enhanced Virtual Karaoke (EVIKA). The EVIKA system consists of two major modules, the facial image enhancement module and the virtual stage simulation module. The facial image enhancement module augments the user'"'"'s image using the embedded Facial Enhancement Technology (F.E.T.) in real-time. The virtual stage simulation module constructs a virtual stage in the display by augmenting the environmental image. The EVIKA puts the user'"'"'s enhanced body image into the dynamic background, which changes according to the user'"'"'s arbitrary motion. During the entire process, the user can interact with the system and select and interact with the virtual objects on the screen. The capability of real-time execution of the EVIKA system even with complex backgrounds enables the user to experience a whole new live virtual entertainment environment experience, which was not possible before.

Claims

1. A method for augmenting visual images of audio-visual entertainment systems, comprising the following steps of:

(a) enhancing facial images of a user or a plurality of users in a video input by superimposing virtual object images to said facial images,

(b) simulating a virtual stage environment image, further comprising steps of processing virtual object image selection, processing music selection, and composing virtual stage images,

(c) setting up masked regions on the simulated virtual stage environment image, and

(d) positioning the masked virtual stage environment image in front of the body image of said user or said plurality of users,

whereby the step for enhancing facial images is processed at the level of local facial features on face images of said user or said plurality of users,

whereby examples of the facial features can be eye, nose, and mouth of said user or said plurality of users, and

whereby the body image of said user or said plurality of users is shown through the transparency channel region of the masked virtual stage environment image.

2. The method according to claim 1, wherein the method further comprises a step for using movement of said user or said plurality of users to trigger dynamically changing virtual background images,

whereby without the movement, said body image of said user or said plurality of users could disappear behind the virtual background images,

whereby this feature adds an interesting and amusing value to the system, in which said user or said plurality of users has to dance as long as said user or said plurality of users wants to see herself/himself on a means for displaying output, and

whereby this feature can be utilized as a method for said user or said plurality of users to participate in a dance in front of the audio-visual entertainment system.

3. The method according to claim 1, wherein the method further comprises a step for attaching musical instrument images, such as a guitar image or a violin image, to said body image of said user or said plurality of users,

whereby the attached musical instrument images dynamically move along with arbitrary motion of said user or said plurality of users in real-time, and

whereby said user or said plurality of users can also play the musical instrument by pretending as if he or she actually plays the musical instrument while looking at the musical instrument image on a means for displaying output.

8. The method according to claim 1, wherein the method further comprises a step for processing the facial image enhancement automatically, dynamically, and in real-time.

9. The method according to claim 1, wherein the step (b) simulating a virtual stage environment image further comprises a touch free interface for processing virtual object image selection and processing music selection.

10. The method according to claim 9, wherein the method further comprises a step for processing

(a) said virtual object image selection and said music selection by said touch free interface,

(b) the enhancement of said facial images at the local facial feature level, and

(c) the composition of the virtual stage images on any arbitrary background in the actual environment rather than a controlled background, such as a blue-screen style background,

whereby the dynamic background construction can be processed by an adaptive background subtraction algorithm.

11. The method according to claim 1, wherein the method further comprises a step for combining the enhanced facial images of said user or said plurality of users and said body image of said user or said plurality of users with dynamically changing virtual background images,

whereby the virtual background images dynamically change according to arbitrary movement of said user or said plurality of users in real-time.

4. An apparatus for augmenting visual images of an audio-visual entertainment system comprising:

(a) one or a plurality of means for capturing facial images from video input image sequences of a user or a plurality of users,

(b) means for displaying output,

(c) means for enhancing said facial images of said user or said plurality of users from said video input image sequences by superimposing virtual object images to said facial images,

(d) means for processing dynamically changing virtual background images according to body movements of said user or said plurality of users,

(e) means for simulating a virtual stage environment image by composing the enhanced facial and body image of said user or said plurality of users, virtual stage images, and virtual objects images, and

(f) means for handling interaction between said user or said plurality of users and said audio-visual entertainment system,

(g) a sound system, and

(h) a microphone,

whereby the means for enhancing facial images processes the facial image enhancement at the level of local facial features on said facial images of said user or said plurality of users, and

whereby examples of the facial features can be eyes, nose, and mouth of said user or said plurality of users.

5. The apparatus according to claim 4, wherein the (c) means for enhancing said facial images of said user or said plurality of users from said video input image sequences further comprises means for using a facial image enhancement process.

6. The apparatus according to claim 4, wherein the (c) means for enhancing said facial images of said user or said plurality of users from said video input image sequences further comprises means for using the embedded FET system for a facial image enhancement process.

7. The apparatus according to claim 4, wherein the (e) means for simulating a virtual stage environment image by composing the enhanced facial and body image of said user or said plurality of users, virtual stage images, and virtual objects images further comprises means for preparing said virtual object images, such as musical instrument images and stage images, off-line.

12. The apparatus according to claim 4, wherein the apparatus further comprises means for enhancing the facial images automatically, dynamically, and in real-time.

13. The apparatus according to claim 4, wherein the means for simulating a virtual stage environment image further comprises means for:

(a) processing virtual object image selection,

(b) processing music selection, and

(c) composing virtual stage images,

wherein the selection is processed by a touch free interface.

14. The apparatus according to claim 4, wherein the apparatus further comprises means

for processing any arbitrary background in the actual environment rather than a controlled background, such as a blue-screen style background,

for constructing said dynamically changing virtual background images,

for processing of said facial images from said user or said plurality of users in order to obtain facial features and body movement information of said user or said plurality of users, and

for processing user interaction by a touch-free interface,

whereby said dynamically changing virtual background images are background images which change according to arbitrary movement of said user or said plurality of users in real-time.

15. The apparatus according to claim 4, wherein the apparatus further comprises a means for combining the enhanced facial images of said user or said plurality of users and the body images of said user or said plurality of users with said dynamically changing virtual background images,

whereby the virtual background images dynamically change according to arbitrary movement of said user or said plurality of users in real-time, and

whereby the enhanced facial images are accomplished at the local facial feature level, such as eyes, nose, and mouth.

16. A method for augmenting images on a means for displaying output of an audio-visual entertainment system, comprising the following steps of:

(a) capturing a plurality of images for a user or a plurality of users with a single or a plurality of means for capturing images,

(b) processing a single image or a plurality of images from the captured plurality of images in order to obtain facial features and body movement information of said user or said plurality of users,

(c) processing selection by said user or said plurality of users for virtual object images on a means for displaying output,

(d) augmenting facial feature images of said user or said plurality of users with the selected virtual object images,

(e) simulating a virtual stage environment image, and

(f) displaying the augmented facial images with said facial feature images of said user or said plurality of users and the simulated virtual stage environment image on said means for displaying output,

whereby the step for augmenting facial feature images is processed at the level of local facial features on face images of said user or said plurality of users,

whereby examples of the local facial features can be eyes, nose, and mouth of said user or said plurality of users, and

whereby the step for augmenting facial feature images of said user or said plurality of users with the selected virtual object images is processed automatically, dynamically, and in real-time.

17. The method according to claim 16, wherein the method further comprises a step for processing touch-free interaction for the selection of said virtual object images.

18. The method according to claim 16, wherein the method further comprises a step for processing music selection by a touch-free interface.

19. The method according to claim 16, wherein the method further comprises a step for processing any arbitrary background in the actual environment rather than a controlled background, such as a blue-screen style background,

for constructing dynamically changing virtual background images,

for processing of said single image or said plurality of images from said captured plurality of images in order to obtain facial features and body movement information of said user or said plurality of users, and

for processing of selection by said user or said plurality of users for said virtual object images on said means for displaying output,

whereby said dynamically changing virtual background images are background images which change according to arbitrary movement of said user or said plurality of users in real-time, and

whereby the system can reside in any arbitrary environment.

20. The method according to claim 19, wherein the method further comprises a step for combining the augmented facial images of said user or said plurality of users and body images of said user or said plurality of users with said dynamically changing virtual background images,

whereby the virtual background images dynamically change according to arbitrary movement of said user or said plurality of users in real-time, and

whereby the augmented facial images are accomplished at the local facial feature level, such as eyes, nose, and mouth.

21. The method according to claim 20, wherein the method further comprises a step for positioning a masked virtual stage image in front of said body images of said user or said plurality of users,

whereby said body images of said user or said plurality of users are shown through the transparency channel region of said masked virtual stage image.

22. The method according to claim 20, wherein the method further comprises a step for using movement of said user or said plurality of users to trigger the dynamically changing background images,

whereby without said movement of said user or said plurality of users, said body images of said user or said plurality of users could disappear behind the background image,

whereby this feature adds an interesting and amusing value to the system, in which said user or said plurality of users have to dance as long as said user or said plurality of users want to see themselves on said means for displaying output, and

whereby this feature can be utilized as a method for said user or said plurality of users to participate in a dance in front of the audio-visual entertainment system.

23. The method according to claim 16, wherein the method further comprises a step for attaching musical instrument images, such as a guitar image or a violin image, to body images of said user or said plurality of users,

whereby the attached musical instrument images dynamically move along with the arbitrary motion of said user or said plurality of users in real-time, and

whereby said user or said plurality of users can also play the musical instrument by pretending as if he or she actually plays the musical instrument while looking at the musical instrument image on said means for displaying output.