Methods and systems for processing an interchange of real time effects during video communication

• US 20070268312A1
• Filed: 05/07/2007
• Published: 11/22/2007
• Est. Priority Date: 05/07/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard, Zalewski, Gary
• Granted Patent: US 8,766,983 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/58 by computing conditions of game characters, e.g. stamina, strength, motivation or energy level
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/65 for computing the condition of a game character
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

Computer implemented methods for interactively modifying a video image are provided. The video image can be transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network. Additionally, the first user and the second user interact through respective computing systems that at least partially execute the computer program. A video capture system is provided that is interfaced with the computer program that can be used to capture real-time video of the first user. The method continues by identifying components of the video image of the first user that can be modified using real-time effects in the captured real-time video. In another operation, the method identifies controller input from either the first user or the second user. The controller input detected by the computing system is identified to determine which of the identified components of the first user will be modified. In response to the identified controller input, another operation of the method augments the real-time video captured of the first user by applying the real-time effects to the identified components of the first user. The method then displays the augmented real-time video of the first user on a screen connected to the computing system of one or both of the first and second users.

Claims

1. A computer implemented method for interactively modifying a video image, the video image transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network and each of the first user and the second user interacting through a respective computing system that is at least partially executing the computer program, comprising:

providing a video capture system interfaced with the computer program;

capturing real-time video of the first user;

identifying components of the video image of the first user that can be modified using real-time effects in the captured real-time video;

identifying controller input from either the first user or the second user, the controller input being detected by the computing system, and the identification of the controller input determines which of the identified components of the first user will be modified;

applying real-time effects to the identified components of the first user in response to the identified controller input, and the real-time video captured of the first user is augmented with the real-time effects; and

displaying augmented real-time video of the first user on a screen connected to the computing system of one or both of the first and second users.

2. The computer implemented method of claim 1, wherein interactively modifying a video image includes:

identification of pixel regions of the video image to identify characteristics of the first user;

tracking the pixel regions over one or more frames; and

applying changes to pixel data contained in the pixel regions so that the video image is interactively modified.

3. The computer implemented method of claim 1, wherein the video capture system includes a camera that captures image frames and digitizes the image frames to define a pixel map of the image frames.

4. The computer implemented method of claim 2, wherein the identified components of the video image relate to the characteristics of the first user.

5. The computer implemented method of claim 4, wherein the characteristics of the first user include facial and body components, and the facial and body components are identified by recognizing characteristics that are common in facial and body components.

6. The computer implemented method of claim 5, wherein a location of eyes of the first user define characteristics that are common facial components.

7. The computer implemented method of claim 1, wherein controller input is defined by user selection of one of button presses, motion, light indicators, relative positional movement, or a combination thereof.

8. The computer implemented method of claim 7, wherein the controller input detected by the computing system is via a wired or wireless link.

9. The computer implemented method of claim 1, where determining which of the identified components of the first user will be modified is assisted by mapping specific controller inputs to the identifying components of the video image.

10. The computer implemented method of claim 2, wherein applying real-time effects to the identified components includes directing the application of changes to the pixel data contained in the pixel regions of the video image, and the real-time effects including one of pre-rendered video animations, themes, custom video animations, and combinations thereof.

11. The computer implemented method of claim 10, wherein when the changes to the pixel data are filtered onto the captured real-time video of the first user, the captured real-time video of the first user is in an augmented state.

12. A computer implemented method for interactively modifying a video image and audio, the video image and audio transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network to enable a chat communication, and each of the first user and the second user interacting through a respective computing system that is at least partially executing the computer program, comprising:

providing a video and audio capture system on each of the respective computing systems of the first and second users, the video and audio capture system being interfaced with the computer program to enable the chat communication;

capturing real-time video and audio of the first user through the video and audio capture system connected to the computing system of the first user;

identifying components of the video image of the first user that can be modified using real-time effects in the captured real-time video;

identifying audio segments of audio captured by the video and audio capture system that can be modified using real-time effects;

identifying user input from either the first user or the second user, and the identification of the user input determines which of the identified audio segments of the first user will be modified;

applying real-time effects to either one or both of the identified components of the first user or the audio segments in response to the identified user input; and

outputting real-time video and audio of the first user on a screen connected to the computing system of one or both of the first and second users, the output real-time video and audio including the applied real-time effects.

13. The computer implemented method of claim 12, wherein interactively modifying a video image includes:

identification of pixel regions of the video image to identify characteristics of the first user;

tracking the pixel regions over one or more frames; and

applying changes to pixel data contained in the pixel regions so that the video image is interactively modified.

14. The computer implemented method of claim 12, wherein the video capture system includes a camera that captures image frames and digitizes the image frames to define a pixel map of the image frames.

15. The computer implemented method of claim 13, wherein applying real-time effects to the identified components includes directing the application of changes to the pixel data contained in the pixel regions of the video image, and the real-time effects including one of pre-rendered video animations, themes, custom video animations, and combinations thereof.

16. A computer implemented method for interactively modifying a video image during chat communication in conjunction with game play over a network, comprising:

providing a video and audio capture system on a respective computing system of the first and second users, the video and audio capture system being interfaced with the computer program to enable the chat communication;

capturing real-time video and audio of a first user through the video and audio capture system connected to the computing system of the first user;

identifying components of the video image of the first user that can be modified using real-time effects in the captured real-time video;

identifying audio segments of audio captured by the video and audio capture system that can be modified using real-time effects;

identifying user input from either the first user or the second user, and the identification of the user input determines which of the identified components audio segments of the first user will be modified;

applying real-time effects to either one or both of the identified components of the first user or the audio segments in response to the identified user input; and

outputting real-time video and audio of the first user on a screen connected to the computing system of one or both of the first and second users, the output real-time video and audio including the applied real-time effects.

17. The computer implemented method of claim 16, wherein interactively modifying a video image includes:

identification of pixel regions of the video image to identify characteristics of the first user;

tracking the pixel regions over one or more frames; and

applying changes to pixel data contained in the pixel regions so that the video image is interactively modified.

18. The computer implemented method of claim 16, wherein the video capture system includes a camera that captures image frames and digitizes the image frames to define a pixel map of the image frames.

19. The computer implemented method of claim 18, wherein user input is defined by user selection of one of button presses, motion, light indicators, relative positional movement, or a combination thereof.

20. The computer implemented method of claim 17, wherein applying real-time effects to the identified components includes directing the application of changes to the pixel data contained in the pixel regions of the video image, and the real-time effects including one of pre-rendered video animations, themes, custom video animations, and combinations thereof.