System and method for automatically editing captured images for inclusion into 3D video game play

• US 6,894,686 B2
• Filed: 05/15/2001
• Issued: 05/17/2005
• Est. Priority Date: 05/16/2000
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation), Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ohno, Takashi, Stamper, Timothy David Joseph, Chesluk, Beau Ner, Stamper, Christopher Timothy John, Teramoto, Noriaki, Edmonds, Mark Alexis, Irwin, Russell William, Koshima, Kazuo, Shimizu, Takao, Tanaka, Hirokazu, Miyamoto, Shigeru, Nishiumi, Satoshi
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/854,942
• Publication Number:
  US 20020082082A1
• Time in Patent Office: 1,463 Days
• Field of Search: 345/419, 345/619, 345/629, 345/473, 345/474, 345/475, 345/423, 345/426, 345/427
• US Class Current: 345/419
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/95 Storage media specially adapted for storing game information, e.g. video game cartridges
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player

Abstract

A video/computer game machine maps external images onto animated game players in a game program. An image of a person'"'"'s face may be mapped onto the head of an animated game player. The person may then play the game using the animated game player having a face that appears to be that of the person. Similarly, images of other persons can be mapped on other animated game players that appear in the video game program. Improved realism of video games is obtained by enabling individuals to create animated game players that have features, e.g., a facial image, of the individual or that were selected and imported by the individual.

Claims

1. A video game system for playing interactive animated video games using hand-operated handheld controls that produces video game animation control signals in response to manipulation by a user, the video game system including a main game unit that generates game images, including at least one animated three-dimensional player object, for animated display during animated game play, the main game unit including a processor and also having a memory that stores at least a portion of a game program, wherein:

the processor receives said game control signals from the handheld controls, said main game unit generating a display of the animated three-dimensional player in response to accessing a memory storing the game program, the game program including an image editor, wherein the editor maps an imported two-dimensional image onto the animated three-dimensional player object, wherein the game program animates said three-dimensional player to move under control of said game control signals the processor receives from the handheld controls,

wherein said processor performs an automatic shade compensation non-linear filter operation that substantially eliminates undesirable shade discontinuities within or alone the edge of the mapped two-dimensional image, said non-linear filter operation modifying the intensity of each pixel in the two-dimensional image as a function of pixel position.

2. A video game system as in claim 1 wherein the imported two-dimensional image is a two-dimensional image of a face.

3. A video game system as in claim 1 further comprising a digital camera coupled to said processor system, and said digital camera captures the two-dimensional image in real time under control of the game program.

4. A video game system as in claim 3 wherein the digital camera is included in a removable cartridge insertable into an insertion port associated with the main game unit.

5. A video game system as in claim 1 wherein the processor comprises a microcontroller and a graphics processor.

6. A video game system as in claim 1 wherein the memory storing the game program and the video editor is included in a game cartridge insertable into an insertion port associated with the main game unit.

7. A video game system as in claim 1 further comprising a player memory electrically coupled to said processor and storing, in the player memory, data indicative of the two-dimensional image as mapped onto the three-dimensional player object.

8. A video game system as in claim 7 wherein the player memory is separable from the main game unit.

9. A video game system as in claim 7 wherein the player memory is a portable removable memory cartridge.

10. A video game system as in claim 9 wherein the player cartridge physically connects to the controls.

11. A video game system as in claim 1 wherein the handheld controls comprise a hand controller separate from the main game unit.

12. A video game system as in claim 1 wherein said handheld controls comprise first and second handheld controllers each having a player cartridge storing data indicative of different two-dimensional images.

13. A video game system as in claim 1 further comprising a display coupled to said main game unit and showing the three-dimensional player object during game play.

14. A video game system as in claim 13 wherein said display is a television.

15. The video game system of claim 1 wherein the image editor provides a coordinate identifier that identifies coordinates on the 2D image to be mapped to triangles for a 3D face.

16. The video game system of claim 1 wherein the image editor is operated by the user of the video game in response to manipulation of said handheld controls.

17. The video game system of claim 1 wherein the image editor allows editing in either a 2D mode or a 3D mode.

18. The video game system of claim 1 wherein the image editor allows editing of a 2D image while displaying the 3D image in real time to show the 3D effects of said editing.

19. The video game system of claim 1 wherein the image editor allows the user to select between plural 3D heads on which to map a 2D image.

20. The video game system of claim 1 wherein the image editor also manipulates a 3D head onto which to map the 2D image, in order improve the appearance of a 2D face mapped onto the head.

21. The video game system of claim 20 wherein the editor provides for 3D head manipulation of both the front view shape and the front-to-back dimension of the head.

22. The video game system of claim 1 further including a portable storage device that stores a representation of a personalized game player.

23. The video game system of claim 1 wherein said image editor maps a 2D facial image onto a 3D head.

24. The video game system of claim 1 wherein the image editor randomly places the two-dimensional image onto computer controlled player objects.

25. The video game system of claim 1 wherein the image editor maps the 2D image onto a selected 3D head.

26. The video game system of claim 1 wherein the handheld controls are disposed on a handheld controller including an insertion slot for receiving a digital camera having a 2D image capture memory disposed therein, said controller insertion slot also receiving a memory for storing a 3D head pre-mapped with the 2D image.

27. The video game system of claim 1 wherein the image editor personalizes a game player to have the face of the user of the video game system.

28. The video came system of claim 1 wherein said video game system includes plural handheld controllers for simultaneous operation by plural associated users each having video game characters associated therewith.

29. The video game system of claim 28 wherein each of said plural video game characters has a personalized face specified by an associated user.

30. The video game system of claim 1 wherein said image editor includes a real time image capture routine that displays a captured image within a template of a predetermined shape.

31. The video game system of claim 30 wherein said template predetermined shape comprises an oval.

32. The video game system of claim 1 wherein the image editor includes a face mapping routine that determines the center of the 2D image relative to a predetermined portion of the three-dimensional player.

33. The video game system of claim 1 wherein the image editor calculates transformed texture coordinates for each of plural vertices of a polygon mesh defining said three-dimensional player object.

34. The video game system of claim 1 wherein the image editor permits the user to edit at least one of the color and the shape of said two-dimensional image by manipulating the handheld controls.

35. The video game system of claim 1 wherein the image editor automatically balances the contrast of the image to reduce unintended effects of shading on the image.

36. The video came system of claim 35 wherein the image editor balances the contrast by comparing the brightness of one side of the image to the brightness of other side of the image, and adjusts contrast in response to results of the comparison.

37. The video game system of claim 36 wherein the image editor applies brightness adjustment linearly across the image without adjusting brightness at the center of the image, in order to avoid creating a perceptible contrast change at the image center.

38. The video game system of claim 1 wherein the image editor applies a non-linear filter to the two-dimensional image so as to modify image intensity as a function of position.

39. In a video game playing system, a method of allowing a video game player to interactively map a two-dimensional image onto a three-dimensional animated object comprising:

(a) obtaining a two-dimensional image;

(b) selecting a three-dimensional object comprising a polygon mesh;

(c) changing, under interactive user control, the shape of the polygon mesh in order to provide a more optimal mapping of the two-dimensional image onto the polygon mesh;

(d) texture mapping the two-dimensional image onto the shape-changed polygon mesh; and

(e) animating, on an interactive real time basis, the three-dimensional object including the texture-mapped two-dimensional image as part of animated video game play,

wherein the method further includes finding the center of the two-dimensional image with respect to the three-dimensional object by defining transformation scalars that convert the position of the three-dimensional polygon mesh into a corresponding two-dimensional texture coordinate to thereby indicate the corresponding position in two-dimensional space that the three-dimensional vertex position should represent.

40. The method of claim 39 wherein the animating step comprises controlling the motion of the three-dimensional object in response to interactive user manipulation of a handheld controller.

41. The method of claim 39 wherein the obtaining step (a) includes capturing the two-dimensional image with a digital camera in real time during video game operation.

42. The method of claim 39 further including repeating said steps (a)-(d) to provide multiplayer game play including plural animated texture-mapped three-dimensional objects interactively controlled by different video game players.

43. The method of claim 39 wherein the shape-changing step comprises changing the dimensions of at least a portion of said object in at least two dimensions.

44. The method of claim 39 wherein the shape-changing step comprises changing the dimensions of at least a portion of said object in three dimensions.

45. The method of claim 39 wherein the two-dimensional image comprises a facial image of an individual having a real-life head shape, and the shape-changing step comprises editing the shape of a three-dimensional virtual head to permit reshaping of the three-dimensional virtual head to better conform to the real-life head shape of the individual.