Game system, game apparatus, storage medium having game program stored therein, and image generation method

• US 9,149,715 B2
• Filed: 02/07/2012
• Issued: 10/06/2015
• Est. Priority Date: 04/11/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Abe, Goro, Fujisawa, Yasuhito
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): CHEN, FRANK S
• Application Number:
  US13/367,808
• Publication Number:
  US 20120264512A1
• Time in Patent Office: 1,337 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/327 using wireless networks, e.g. Wi-Fi® or piconet
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images

Abstract

An example game system includes a game apparatus and a terminal device capable of communicating with the game apparatus. The terminal device transmits a pickup image obtained by an image pickup section to the game apparatus. The game apparatus receives the pickup image, and generates a first image by synthesizing the pickup image at least in part with an image of a predetermined virtual space as viewed from a first virtual camera. The first image is outputted and displayed on a predetermined display device. In addition, the game apparatus generates a second image of the virtual space as viewed from a second virtual camera different from the first virtual camera. The second image is outputted and displayed on the terminal device.

Claims

1. A game system comprising:

a terminal device that includes;

a camera configured to obtain an image;

a transceiver configured to transmit the image obtained by the camera; and

a display device configured to display images; and

a game apparatus that includes;

a transceiver configured to receive the image transmitted by the transceiver of the terminal device; and

a processing system that includes processing circuitry coupled to a memory, the processing system configured to;

position at least a part of the obtained image that is received by the game apparatus on a predetermined plane that is located within a three-dimensional virtual space;

locate a first virtual camera in the three-dimensional virtual space so as to view at least the part of the obtained image that is located in the three-dimensional virtual space;

generate a first image in accordance with the located first virtual camera, the first image including a representation of the part of the obtained image located in the three-dimensional virtual space;

output the generated first image to a display device that is different from the display device of the terminal device;

generate a second image of the three-dimensional virtual space as viewed from a second virtual camera different from the first virtual camera, where the first and second virtual cameras are located on opposite sides of, and directed towards, the predetermined plane that includes the part of the obtained image, where the part of the obtained image is viewable from the located first virtual camera but is not viewable from the second virtual camera; and

transmit, via the transceiver of the game apparatus, the generated second image to the terminal device, wherein

at least the part of the obtained image is rendered on the predetermined plane, thereby generating the first image, and

in accordance with generation of the second image of the three-dimensional virtual space, the virtual space is changed so that the predetermined plane is not present or is made transparent for generation of the second image.

2. The game system according to claim 1, wherein the at least the part of the obtained image is located at a position of the predetermined plane.

3. The game system according to claim 1, wherein the processing system of the game apparatus is further configured to:

arrange a rendering plane as the predetermined plane within the three-dimensional virtual space, the predetermined plane having at least the part of the obtained image rendered thereon,

the generated first image includes the rendering plane.

4. The game system according to claim 3, wherein the first virtual camera is located at a position and an attitude in the three-dimensional virtual space such that the rendering plane is viewed from a side on which the part of the obtained image is rendered.

5. The game system according to claim 3, wherein the second virtual camera is located at a position and an attitude in the three-dimensional virtual space such that the rendering plane is viewed from an opposite side to a side on which the part of the obtained image is rendered.

6. The game system according to claim 3, wherein the processing system of the game apparatus is further configured to execute game processing on the basis of an action of a virtual object arranged in the three-dimensional virtual space and the part of the obtained image rendered on the rendering plane.

7. The game system according to claim 6, further comprising an operating device separate from the terminal device and the game apparatus, the operating device configured to communicate operation data to the game apparatus,

wherein the processing system of the game apparatus is further configured to control the action of the virtual object in accordance with the communicated operation data.

8. The game system according to claim 1, processing system of the game apparatus is further configured to:

perform image recognition processing on the obtained image, or a part thereof; and

perform game control processing to add a change to the three-dimensional virtual space on the basis of the result for the performed image recognition processing.

9. The game system according to claim 1, wherein,

the terminal device further includes a microphone,

the transceiver of the terminal device is further configured to transmit data based on sound detected by the microphone to the game apparatus, and

processing system of the game apparatus is further configured to perform game control processing to add a change to the three-dimensional virtual space on the basis of the sound data.

10. The game system according to claim 1, wherein,

the transceiver of the terminal device is further configured to transmit operation data, which represents an operation performed by a player using the terminal device, to the game apparatus, and

the processing system of the game apparatus is further configured to perform game control processing to add a change to the three-dimensional virtual space on the basis of the operation data.

11. The game system according to claim 10, wherein,

the terminal device further includes a sensor unit that is configured to sense motion of the terminal device and output motion data representing information about the motion of the terminal device,

wherein the transmitted operation data includes the motion data.

12. The game system according to claim 10, wherein,

the terminal device further includes a position detector configured to detect an input position on a screen of the display device, and

the transmitted operation data further includes data based on the detected input position.

13. The game system according to claim 10, wherein,

the terminal device further includes at least one operation member, and

the transmitted operation data includes data based on detected actuation of the at least one operation member.

14. An image generation method to be executed in an information processing system including an information processing apparatus, which includes a transceiver and a processing system that includes processing circuitry coupled to a memory device, and a terminal device capable of communicating with the information processing apparatus, the terminal device including a camera configured to acquire a taken image, a transceiver, and a display device, the method comprising:

receiving, via the transceiver of the information processing apparatus, an image that is based on the taken image obtained by the camera of the terminal device;

positioning, by using the processing system of the information processing apparatus, at least a part of the obtained image received by the game apparatus in a three-dimensional virtual space;

locating, by using the processing system of the information processing apparatus, a first virtual camera in the three-dimensional virtual space so as to view at least the part of the obtained image that is located in the three-dimensional virtual space;

generating, by using the processing system of the information processing apparatus, a first image in accordance with the located first virtual camera, the first image including a representation of the part of the obtained image located in the three-dimensional virtual space;

outputting the generated first image to a display device that is different from the display device of the terminal device;

generating a second image of the three-dimensional virtual space as viewed from a second virtual camera different from the first virtual camera, wherein the part of the taken obtained image that is positioned within the three-dimensional virtual space is not viewable in the second image generated via the second virtual camera; and

transmitting, via the transceiver of the information processing apparatus, the generated second image to the terminal device, wherein

at least the part of the obtained image is rendered on onto a predetermined plane that is positioned within the three-dimensional virtual space at a position therein, thereby generating the first image, and

for generation of the second image where the second virtual camera is orientated to view the position at which the predetermined plane is located within the three-dimensional virtual space, the virtual space is changed so that the predetermined plane is not present or is made transparent for generation of the second image.

15. The image generation method according to claim 14, wherein the first and second virtual cameras obtain their respective images of the predetermined plane within the three-dimensional virtual space from opposite sides.

16. The image generation method according to claim 15, wherein at least the part of the obtained image is located at a position of the predetermined plane.

17. The image generation method according to claim 14, further comprising:

locating a rendering plane, in the three-dimensional virtual space, on which to render at least the part of the obtained image.

18. The image generation method according to claim 17, wherein the first virtual camera is located at a position and an attitude in the three-dimensional virtual space set such that the rendering plane is viewed from a side on which the part of the obtained image is rendered.

19. The image generation method according to claim 17, wherein the second virtual camera is located at a position and an attitude in the three-dimensional virtual space such that the position of the rendering plane is viewed from an opposite side to a side on which the part of the obtained image is rendered.

20. The image generation method according to claim 17, further comprising:

executing, by using the processing system of the information processing apparatus, game processing on the basis of an action of a virtual object arranged in the three-dimensional virtual space and the obtained image rendered on the rendering plane.

21. The image generation method according to claim 20, wherein the information processing system further includes an operating device, which is separate from the information processing apparatus and the terminal device, configured to communicate with the information processing apparatus,

wherein the game processing includes controlling the virtual object in accordance with an operation on the operating device.

22. The image generation method according to claim 14, further comprising:

performing image recognition processing on the obtained image, and

performing game control processing to add a change to the three-dimensional virtual space on the basis of a result from the performed image recognition processing.

23. The image generation method according to claim 14, wherein the terminal device further includes a microphone, the method further comprising:

receiving, via the transceiver of the information processing apparatus, sound data based on the sound detected by the microphone, and

performing game control processing to add a change to the three-dimensional virtual space on the basis of the sound data.

24. The image generation method according to claim 14, further comprising:

receiving, via the transceiver of the information processing apparatus, operation data representing an operation performed by a player on the terminal device;

performing game control processing to add a change to the three-dimensional virtual space on the basis of the operation data.

25. The image generation method according to claim 24, wherein the terminal device further includes a motion sensor configured to output motion data based on sensed motion, the method further comprising:

receiving the motion data.

26. The image generation method according to claim 24, wherein the terminal device further includes a position detection unit capable of detecting an input position on a screen of the display unit, the method further comprising:

receiving data based on the position detected by the position detection unit.

27. The image generation method according to claim 24, wherein the terminal device further includes at least one operation member, the method further comprising:

receiving operation data representing operation on the operation member.

28. A game apparatus comprising:

a transceiver configured to receive an image from another computer apparatus that includes (a) a camera that has obtained the image and (b) a display device;

a processing system that includes at least one processing circuit coupled to a memory and the transceiver, the processing system configured to;

position at least a part of the obtained image that is received by the game apparatus in a three-dimensional virtual space;

locate a first virtual camera in the three-dimensional virtual space so as to view at least the part of the obtained image that is located in the three-dimensional virtual space;

generate a first image in accordance with the located first virtual camera, the first image including a representation of the part of the obtained image located in the three-dimensional virtual space;

output the generated first image to a different display device that is separate from the display device of the another computer apparatus;

generate a second image of the three-dimensional virtual space as viewed from a second virtual camera different from the first virtual camera, where the first and second virtual cameras are located on opposite sides of, the location in the three-dimensional virtual space where the part of the obtained image is located such that the location is viewable by both the first and second virtual cameras, where the representation of the part of the image is included in the first image, but no representation of the part of the image is included in the second image that is generated with the second virtual camera; and

transmit, via the transceiver of the game apparatus, the generated second image to the another computer apparatus for display on the display device thereon, wherein

at least the part of the obtained image is rendered onto a virtual model that is positioned within the three-dimensional virtual space at the location, thereby generating the first image, and

in generating the second image, the virtual model, onto which at least the part of the obtained image is rendered for the first image, is removed from three-dimensional virtual space or set as transparent such that virtual space locations located beyond the location at which the virtual model was located are viewable from the second virtual camera and visually included in the generated second image.

29. A non-transitory computer-readable storage medium having stored therein a game program to be executed by a computer system of a game apparatus, the game apparatus including a transceiver configured to communicate with a terminal device that includes a camera and a display device, the computer system including at least one processing circuit coupled to a memory device, the game program comprising instructions that cause the computer to:

accept, via the transceiver, a pickup image that has been transmitted by the terminal device to the transceiver of the game apparatus, the pickup image being obtained by the camera of the terminal device;

position at least a part of the pickup image that is received by the game apparatus in a three-dimensional virtual space;

locate a first virtual camera in the three-dimensional virtual space so as to view at least the part of the pickup image that is located in the three-dimensional virtual space;

generate a first image in accordance with the located first virtual camera, the generated image being intended to be displayed on a predetermined display device as a first image, the first image including a representation of the part of the pickup image located in the three-dimensional virtual space;

output the generated first image to a display device that is different from the display device of the terminal device;

generate a second image of the three-dimensional virtual space as viewed from a second virtual camera different from the first virtual camera, where the first and second virtual cameras are located on opposing sides of, and directed towards, the position in the three-dimensional virtual space where the part of the pickup image is located, where the part of the pickup image is viewable from the first virtual camera but is not viewable from the second virtual camera; and

transmit, via the transceiver, the generated second image to the terminal device for display on the display device, wherein

at least the part of the pickup image is rendered on a predetermined plane that is located at the position within the three-dimensional virtual space, thereby generating the first image, and

for generation of the second image, the three-dimensional virtual space regarding the predetermined plane is changed so that the predetermined plane is not present or is transparent for generation of the second image.