Game system, game device, storage medium storing game program, and image generation method

• US 8,747,222 B2
• Filed: 02/08/2012
• Issued: 06/10/2014
• Est. Priority Date: 06/02/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Yamashita, Yoshikazu
• Primary Examiner(s): Shah, Milap
• Application Number:
  US13/368,826
• Publication Number:
  US 20120309523A1
• Time in Patent Office: 853 Days
• Field of Search: 463 30- 32, 463 36- 43, 463/46, 463/47
• US Class Current: 463/31
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6669 using a plurality of virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character change rooms
• CPC Class Codes: A63F 2300/6684 by dynamically adapting its position to keep a game object in its viewing frustrum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player

Abstract

An example game system includes a portable display that includes an image-capturing unit, and a display unit. The game device obtains first operation data representing an operation performed on a controller device, second operation data representing an operation performed on the portable display, and data of a image captured by the image-capturing unit. In a virtual space, an action of a first control object is controlled based on first operation data and an action of a second control object is controlled based on second operation data. The game device generates a first game image including a game image that represents the virtual space and corresponds to the first control object, and at least a portion of the captured image. The first game image is displayed on a predetermined display. The game device generates a second game image corresponding to the second control object, and displays the second game image.

Claims

1. A game system comprising:

a plurality of controller devices;

a portable display device that includes a camera and a display; and

a game device comprising;

a processing arrangement that includes at least one processor coupled to a memory device, the processing arrangement coupled to the plurality of controller devices and the portable display device, the processing arrangement configured to;

obtain respective first operation data representing operations performed on the plurality of controller devices;

obtain second operation data representing an operation performed on the portable display device; and

obtain data of a captured image captured by the camera;

control a plurality of first control objects that are located in a virtual space based on the respective first operation data; and

control an action of a second control object that is located in the virtual space based on the second operation data;

generate a first game image that includes a plurality of different game images respectively corresponding to the plurality of first control objects and at least a portion of the captured image;

generate a second game image that includes the second control object located in the virtual space;

output the first game image to a predetermined display device that is separate from the portable display device; and

output the second game image to the portable display device.

2. The game system according to claim 1, wherein the processing arrangement is further configured to:

perform a recognition process for recognizing a face in the captured image; and

generate the first game image so that the first game image includes a portion of the captured image that includes an image of the recognized face.

3. The game system according to claim 2, wherein the processing arrangement is further configured to perform a predetermined image process on the image of the captured image that includes the recognized face to thereby generate the first game image that includes the processed image.

4. The game system according to claim 2, wherein the processing arrangement is further configured to:

perform the recognition process in response to obtaining the captured image;

generate a first game image including the image of the recognized face in response to the recognition of the face by the recognition process; and

output the first game image to the predetermined display device in response to the generation of the first game image.

5. The game system according to claim 4, wherein the processing arrangement is further configured to:

successively obtain data of the captured image; and

update the image of the recognized face with a frequency that is less than that with which the data of the captured image is obtained.

6. The game system according to claim 1, wherein the processing arrangement is further configured to:

set plural first virtual cameras respectively corresponding the plurality of first control objects;

set a second virtual camera corresponding to the second control object in the virtual space;

generate the plurality of game images based on views of the virtual space that respectively correspond to the plural first virtual cameras; and

generate the second game image based on a view of the virtual space from the second virtual camera.

7. The game system according to claim 6, wherein the processing arrangement is further configured to:

move the plurality of control objects in the virtual space based on the first operation data; and

set the plural virtual cameras so that the respectively corresponding plurality of first virtual objects are in a field of view.

8. The game system according to claim 6, wherein the processing arrangement is further configured to:

move a second object arranged in the virtual space, as the second control object, based on the second operation data; and

set the second virtual camera so that the second object is included in a viewing field range of the second virtual camera.

9. The game system according to claim 1, wherein the processing arrangement is further configured to:

store a captured image obtained at a predetermined point in time in a memory device; and

a first game image is generated to include at least a portion of the captured image stored in the memory device.

10. The game system according to claim 1, wherein the processing arrangement is further configured to:

determine whether a predetermined game condition has been satisfied for each of the plurality of first control objects; and

generate a first game image by superimposing at least a portion of the captured image onto each one of the different game images that correspond to those of the plurality of first control objects for which the predetermined game condition has been satisfied.

11. A game system comprising:

a portable display device that includes a camera and a display; and

a game device that is separately provided from the portable display device, the game device comprising;

a transceiver that is configured to receive operation data representing an operation performed on the portable display device, and a captured image captured by the camera; and

a processing system that includes at least one processor coupled to a memory, the processing system configured to;

control an action of a virtual object that is located in a virtual world based on the operation data;

generate a first game image which includes;

     1) an image of a view of the virtual world that includes the virtual object, and

     2) at least some of the captured image;

generate a second game image that represents the virtual space and corresponds to the virtual object;

output the first game image to a predetermined display device separate from the portable display device; and

output, using the transceiver, the second game image to the portable display device.

12. A method for use with a game system that comprises a plurality of controller devices, a portable display device, and a game device, wherein the portable display device includes a camera and a display device, the method comprising:

obtaining respective first operation data representing operations performed by the plurality of controller devices;

obtaining second operation data representing an operation performed on the portable display device;

obtaining data of a captured image that has been captured by the camera;

controlling, using at least one processor of the game system, each one of a plurality of first control objects that are each located in a virtual space based on respective application of first operation data that represent operations performed by the plurality of controller devices;

controlling, using the at least one processor of the game system, an action of a second control object located in the virtual space based on the second operation data;

generating, using the at least one processor of the game system, a first game image that includes

     1) a plurality of game images that respectively correspond to the plurality of first control objects that are located in the virtual space, and

     2) at least a portion of the captured image;

generating, using the at least one processor of the game system, a second game image that corresponds to the second control object located in the virtual space;

displaying the first game image on a predetermined display device that is separate from the portable display device; and

displaying the second game image on the display device of the portable display device.

13. The method according to claim 12, further comprising:

performing, using the at least one processor of the game system, a recognition process of recognizing a face in the captured image, wherein;

the portion of the captured image that is included in the first image includes the face recognized by the recognition process.

14. The method according to claim 13, wherein a predetermined image process is performed on the image of the captured image that includes the face recognized by the recognition process, thereby generating the first game image including the processed image.

15. The method according to claim 13, wherein:

the recognition process is performed in response to the obtainment of the captured image;

the first game image includes the image of the recognized face in response to the recognition of the face by the recognition process; and

the first game image is displayed on the display device in response to the generation of the first game image.

16. The method according to claim 15, wherein:

data of the captured image is obtained successively; and

the image of the recognized face is updated with a frequency that is less than that with which data of the captured image is obtained.

17. The method according to claim 12, further comprising:

setting a plurality of first virtual cameras that respectively correspond to the plurality of first control objects in the virtual space; and

setting a second virtual camera that corresponds to second control object in the virtual space, wherein;

the plurality of game images are generated based on a view of the virtual space from respective first virtual cameras that respectively correspond to the plurality first control objects; and

the second game image is generated based on a view of the virtual space from the second virtual camera.

18. The method according to claim 17, wherein:

each one of the plurality of first control objects moves through the virtual space based on respective application of the first operation data; and

each one of the plurality of first virtual cameras is set so that the respectively corresponding first control object is maintained within a viewing field of the respective first virtual camera.

19. The method according to claim 17, wherein:

the second control object is controlled to move through the virtual space based on the second operation data; and

the second virtual camera is set so that the second object is maintained within the view of the virtual space of the second virtual camera.

20. The method according to claim 12, further comprising:

storing a captured image obtained at a predetermined point in time in a predetermined storage device, wherein;

the portion of the captured image that is included in the first game is based on stored captured image.

21. The method according to claim 12, further comprising:

determining whether a predetermined game condition has been satisfied for each one of the plurality of first control objects, wherein;

the first game is generated by superimposing at least a portion of the captured image onto at least one of the plurality of game images that corresponds one of the plurality of first control objects for which the predetermined game condition has been satisfied.

22. A method for use with a game system comprising a portable display device and a game device that is separate from the portable display device, the portable display device comprising a camera and a display device, the game device comprising a processing system that includes at least one processor coupled to a memory and a transceiver that is configured to communicate with the portable display device, the method comprising:

obtaining, using the transceiver of the game device, operation data representing an operation performed on the portable display device and a captured image captured by the camera;

controlling, using the processing system of the game device, an action of a predetermined control object based on the operation data in a virtual space;

generating, using the processing system of the game device, a first game image that includes

     1) a game image that represents the virtual space and corresponds to the control object, and

     2) at least a portion of the captured image;

generating, using the processing system of the game device, a second game image that represents the virtual space and corresponds to the control object;

displaying the first game image on a predetermined display device separate from the portable display device; and

displaying the second game image on the display device of the portable display device.

23. A game device capable of communicating with a plurality of controller devices and a portable display device, the game device comprising:

a processing system that includes at least one processor, the processing system configured to;

receive first operation data that represents operations performed on each one of the plurality of controller devices;

receive second operation data representing an operation performed on the portable display device;

receive data of a captured image captured by a camera of the portable display device;

control plural first control objects that are each located in a virtual space based on the first operation data;

control an action of a second control object located in a virtual space based on the second operation data;

generate a first game image which includes (1) plural game images that respectively correspond to the plural first control objects located in the virtual space and corresponds to the first control object and (2) at least a portion of the captured image;

generate a second game image that represents the virtual space and corresponds to the second control object;

cause the first game image to be output to a display device separate from the portable display device; and

cause the second game image to be output to the portable display device.

24. A game device capable of communicating with a portable display device that is separate from the game device, the game device comprising:

a data obtaining unit that is, by using at least one processor of the game device, configured to obtain operation data representing an operation performed on the portable display device and a captured image captured by an image-capturing unit of the portable display device;

an action control unit that is, by using the at least one processor of the game device, configured to control an action of a control object that is located in a virtual space based on the operation data;

a first image generation unit that is, by using the at least one processor of the game device, configured to generate a first game image which includes (1) a game image that corresponds to the control object in the virtual space, and (2) at least a portion of the captured image;

a second image generation unit that is, by using the at least one processor of the game device, configured to generate a second game image that corresponds to the control object located in the virtual space;

a first image output unit that is, by using the at least one processor of the game device, configured to output the first game image to a predetermined display device separate from the portable display device; and

a second image output unit that is, by using the at least one processor of the game device, configured to output the second game image to the portable display device.

25. A non-transitory computer-readable storage medium storing a game program to be executed by a computer of a game device capable of communicating with plural controller devices and a portable display device, the game program comprising instructions that cause the computer to:

acquire first operation data representing plural operations respectively performed on plural controller devices;

acquire second operation data representing an operation performed on the portable display device;

acquire data of a captured image that has been captured by a camera of the portable display device;

control actions for plural control objects, each of which are located in a virtual space, based on the acquired first operation data;

control an action of a second control object that is located in the virtual space based on the second operation data;

generate a first game image to be displayed on a display device that is separate from the portable display device, the first game image including (1) plural game images being divided into separate areas of the first game image, where each of the plural game images represents a corresponding one of the plural control objects that are located in the virtual space, and (2) at least a portion of the captured image; and

generate a second game image to be displayed on the portable display device, the second game image representing the second control object in the virtual space.

26. A non-transitory computer-readable storage medium storing a game program to be executed by a computer of a game device capable of communicating with a portable display device that is separate from the game device, the game program comprising instructions that cause the computer to:

obtain operation data representing an operation performed on the portable display device, and a captured image captured by a camera of the portable display device;

control an action of a predetermined control object that is located in a virtual space based on the operation data;

generate a first game image to be displayed on a predetermined display device that is separate from the portable display device, the first game image including (1) a game image of the control object that is located in the virtual space as viewed from a virtual camera whose pose is maintained in correspondence to the control object, and (2) at least a portion of the captured image; and

generate a second game image to be displayed on the portable display device, the second game image representing the virtual space and corresponding to the control object.

27. A video game system that provides common viewing to a virtual space for a plurality of players, the video game system comprising:

a plurality of controllers that are each configured to provide input to control aspects of a respective video game object located in the virtual space;

a portable game device comprising;

at least one control input configured to acquire input based on player use of the portable game device;

a display device configured to display a view the virtual space that includes a virtual object that is controlled, at least in part, in accordance with the acquired input of the at least one control input of the portable game device; and

at least one camera configured to acquire a captured image while the display device is displaying the view of the virtual space; and

a common display configured to, in accordance with video game images generated by at least one processor of the video game system, simultaneously display;

a plurality of separate viewpoints of the virtual space that respectively correspond to the video game objects that are controlled, at least in part, based on respective inputs provided from the plurality of controllers; and

a further image that is based on the captured image.

28. The video game system of claim 27, wherein the plurality of separate viewpoints are adjusted in accordance with movement of a respective video game object within the virtual world.

29. A video game system that provides viewing of a virtual space that includes a first virtual object and a second virtual object, the video game system comprising:

a video game controller configured to provide input to control aspects of the first virtual object that is located in the virtual space;

a portable game device comprising;

at least one input configured to detect player operations of the portable game device;

a display device configured to display a second view the virtual space, where the second view includes the second virtual object, where the second virtual object is controlled, at least in part, in accordance with the detected player operations of the portable game device; and

at least one camera configured to acquire a captured image while the display device is displaying the second view of the virtual space; and

a common display configured to, in accordance with video game images generated by at least one processor of the video game system, simultaneously display;

a first view of the virtual space that is dependent on how the first virtual object is controlled in the virtual space using, at least in part, the input provided from the video game controller; and

a further image that is based on the captured image.

30. The video game system of claim 29, wherein the at least one input includes at least one of:

an acceleration sensor, a gyro sensor, and an operation section.

31. A game system comprising:

a game controller configured to be operated by a first user to control a first object that is located in a virtual space;

a portable device configured to be operated by a second user, the portable device comprising;

a first transceiver;

a display configured to display a second game image of the virtual space that includes a second virtual object that is controlled, at least in part, based on operation of the portable device by the second user; and

at least one camera configured to capture an image in correspondence with operation of the portable device for controlling the second virtual object; and

a game device comprising;

a second transceiver configured to communicate with the first transceiver of the portable device; and

a processing system that includes at least one processor, the processing system configured to;

control the first object located in the virtual space based on operation inputs provided from the game controller;

generate a first game image that includes

     1) an image of the virtual space with a view of the first object that is set in accordance with a position and/or direction of the first object and

     2) another image that is at least a portion of the captured image or an image that is a function of the captured image; and

output the generated first game image to a display device that is different from the display of the portable device.