Methods for interactive communications with real time effects and avatar environment interaction

• US 20070260984A1
• Filed: 05/07/2007
• Published: 11/08/2007
• Est. Priority Date: 05/07/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard, Zalewski, Gary
• Granted Patent: US 8,601,379 B2
• Time in Patent Office: Days
• US Class Current: 715/706
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/58 by computing conditions of game characters, e.g. stamina, strength, motivation or energy level
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/65 for computing the condition of a game character
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

Computer implemented method for interactively animating an avatar in response to real world input are provided. The avatar can be transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network. Additionally, the first user and the second user each interact using a respective computing system that is at least partially executing the computer program. The method is initiated by identifying components of the avatar representing the first user that can be modified using real-time effects. The method continues by the computing system identifying controller input from either the first user or the second user. The identification of the controller input is used to determine which of the identified components of the avatar representing the first user will be modified. In response to the identified controller input the real-time effects are applied to the identified components of the avatar that represent the first user. The avatar of the first user augmented to reflect the application of the real-time effects. In another operation, the method displays the augmented avatar of the first user on a screen connected to the computing system of one or both of the first and second users.

Claims

1. A computer implemented method for interactively animating an avatar in response to real world input, the avatar transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network and each of the first user and the second user interacting through a respective computing system that is at least partially executing the computer program, comprising:

identifying components of the avatar representing the first user that can be modified using real-time effects;

identifying controller input from either the first user or the second user, the controller input being detected by the computing system, and the identification of the controller input determines which of the identified components of the avatar representing the first user will be modified;

applying the real-time effects to the identified components of the avatar representing the first user in response to the identified controller input, the avatar of the first user being augmented with the real-time effects; and

displaying the augmented avatar of the first user on a screen connected to the computing system of one or both of the first and second users.

2. The computer implemented method of claim 1, wherein interactively modifying an avatar image includes:

identification of pixel regions of the avatar to identify characteristics of the avatar representing the first user;

tracking the pixel regions over one or more frames; and

applying changes to pixel data contained in the pixel regions so that the avatar is interactively modified.

3. The computer implemented method as described in claim 1, further comprising:

providing a video capture system interfaced with the computer program;

capturing real-time video of the first user;

processing the real-time video to identify at least one facial expression of the first user;

periodically updating a facial expression of the avatar representing the first user to correspond with the facial expression of the first user.

4. The computer implemented method of claim 3, wherein the video capture system includes a camera that captures image frames and digitizes the image frames to define a pixel map of the image frames.

5. The computer implemented method of claim 4, wherein a location of eyes of the first user are used to determine facial characteristics that define the facial expression of the first user.

6. The computer implemented method of claim 1, wherein controller input is defined by user selection of one of button presses, motion, light indicators, relative positional movement, or a combination thereof.

7. The computer implemented method of claim 6, wherein the controller input detected by the computing system is via a wired or wireless link.

8. The computer implemented method of claim 1, where determining which of the identified components of the first user will be modified is assisted by mapping specific controller inputs to the identifying components of the avatar.

9. The computer implemented if claim 1, further comprising:

providing an audio capture system interfaced with the computer program;

capturing real-time audio of the ambient sounds within an environment associated with the first user;

processing the ambient sounds to identify particular sounds;

automatically applying real-time effects to avatars representing the first user or the second user in response to the identified particular sounds within the ambient sounds.

10. The computer implemented method as described in claim 9, wherein the audio capture system includes an array of unidirectional microphones that capture sounds from different portions of the environment associated with the first user.

12. The computer implemented method of claim 10, wherein the audio capture system includes a microphone that captures vocal sounds from the first user and digitizes the vocal sounds to define a vocal characteristic of the first user.

13. The computer implemented method of claim 10, wherein the video capture system includes a camera that captures image frames and digitizes the image frames to define a pixel map of the image frames.

14. The computer implemented method of claim 13, wherein the detecting real-time changes in facial expression includes:

identifying pixel regions of the image frames to identify facial characteristics of the first user; and

tracking and comparing the pixel regions over one or more frames.

11. A computer implemented method for automatically modifying an avatar image in substantial real-time in conjunction with communication over a network, comprising:

providing a video and audio capture system on a respective computing system of a first and a second users, the video and audio capture system being interfaced with the computer program to enable the real-time communication;

detecting real-time changes in facial expression of the first user in the captured video of the first user;

detecting real-time changes in vocal characteristics of the first user;

automatically applying real-time effects to the avatar image representing the first user in response to the monitored real-time video and audio of the first user; and

outputting the avatar image representing the first user with the automatically applied real-time effect on a screen connected to the computing system of one or both of the first and second users.

15. The computer implemented method of claim 11, wherein detecting real-time changes in vocal characteristics includes:

capturing a first vocal characteristic of the first user at a first time;

capturing a second vocal characteristic of the first user at a second time;

comparing the first vocal characteristic to the second vocal characteristic; and

determining a magnitude for the real-time effects.

16. The computer implemented method of claim 15, further comprising:

applying the real-time effects with the magnitude either progressively over time or at once.

17. The computer implemented method of claim 16, wherein when the real-time effects are applied progressively over time, a characteristic of the avatar changes is size, shape, or expression, or a combination thereof.

18. A computer readable media including program instructions for automatically modifying an avatar image in substantial real-time in conjunction with communication over a network, the computer readable media, comprising:

program instructions for providing a video and audio capture system on a respective computing system of a first and a second users, the video and audio capture system being interfaced with the computer program to enable the real-time communication;

program instructions for detecting real-time changes in facial expression of the first user in the captured video of the first user;

program instructions for detecting real-time changes in vocal characteristics of the first user;

program instructions for automatically applying real-time effects to the avatar image representing the first user in response to the monitored real-time video and audio of the first user; and

program instructions for outputting the avatar image representing the first user with the automatically applied real-time effect on a screen connected to the computing system of one or both of the first and second users, and detecting real-time changes in vocal characteristics includes, processing program instructions for directing operations of,

(i) capturing a first vocal characteristic of the first user at a first time;

(ii) capturing a second vocal characteristic of the first user at a second time;

(iii) comparing the first vocal characteristic to the second vocal characteristic; and

(iv) determining a magnitude for the real-time effects.

19. The computer readable media as recited in claim 18, wherein the program instructions are at least partially stored on one of the computing systems of the first and second user and on a computing system coupled to the network.

20. The computer readable media as recited in claim 18, further comprising:

program instructions for applying the real-time effects with the magnitude either progressively over time or at once, and when the real-time effects are applied progressively over time, a characteristic of the avatar changes is size, shape, or expression, or a combination thereof.