Computer-readable storage medium having stored therein display control program, display control apparatus, display control system, and display control method

• US 8,952,956 B2
• Filed: 02/17/2011
• Issued: 02/10/2015
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Sato, Katsuhisa, Nakamura, Masahiro
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Du, Haixia
• Application Number:
  US13/029,282
• Publication Number:
  US 20120154377A1
• Time in Patent Office: 1,454 Days
• Field of Search: 345/419
• US Class Current: 345/419
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/8058 Virtual breeding, e.g. tamagotchi
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

A game apparatus acquires a designated position Q, on an image, which corresponds to a touch position T detected by a touch panel, and a depth value of the designated position Q. A game apparatus calculates a designated three-dimensional position P in a three-dimensional virtual space, based on the designated position Q on the image and the depth value. The game apparatus next determines an orientation of a cursor object to arrange the cursor object at the calculated designated three-dimensional position P. The game apparatus next uses a virtual camera to take an image of the virtual space including the cursor object, and displays the image on an upper LCD.

Claims

1. A non-transitory computer-readable storage medium having stored therein a display control program having computer-executable code or instructions which, when executed by one or more processors of a computerized display control, causes the display control to function and perform operations comprising:

generating an image in which depth values are set, by taking a two-dimensional image of a three-dimensional virtual space by a virtual camera and setting depth values that are associated with each pixel position of the image;

acquiring an associated depth value for a pixel of the image corresponding to a designated position on the image, wherein the designated position on the image is provided by a user via an input touch position designation;

calculating, based on a depth value associated to the designated position on the image, a spatial position in a depth direction in the virtual space as viewed from the virtual camera;

producing a virtual designating object at the calculated spatial position in the depth direction in the virtual space; and

displaying, on a first display area, a first image taken of the virtual space which includes the virtual designating object produced therein and displaying a second image on a second display area, the second image being based upon the first image, wherein the input touch position designation is provided via a second display area, which is different from the first display area and which is capable of detecting a touch position on the second display area, and wherein the depth value is acquired based upon the touch position and a corresponding pixel position in the second image displayed on the second display area.

2. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 1, wherein

the calculated spatial position corresponds to a designated three-dimensional position in the virtual space, based on both the designated position on the image and the depth value of the image at the designated position, and wherein the display control program causes the display control to perform operations comprising:

producing the virtual designating object at the designated three-dimensional position in the virtual space.

3. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

the designated three-dimensional position in the virtual space is calculated based on a location of the designated position on the two-dimensional image in up-down and left-right directions in the virtual space as viewed from the virtual camera and the depth value associated with the designated position.

4. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

the designating object is a cursor object indicating a position within the virtual space.

5. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 4, wherein the display control program causes the display control to perform operations comprising:

producing a second virtual object to be displayed in the virtual space,

generating an image by taking, via the virtual camera, an image of the virtual space including the second object, and

in the case where the designated position exists in an area in which the second object is displayed, determining, based on the designated three-dimensional position, an orientation of the designating object, and arranging in the virtual space the designating object in the determined orientation.

6. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 5, wherein

the second object is formed of a plurality of parts,

the display control program further causes one or more processors of the display control to function and perform operations comprising:

determining, based on the designated position, a designated part from among the plurality of parts of the second object, and

determining the orientation of the designating object, according to the designated part, and arranging the designating object in the virtual space in the determined orientation.

7. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 6, wherein

arranging an object further includes determining an orientation of the designating object, according to a normal line of the designated part at the designated three-dimensional position.

8. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 7, wherein

each of the plurality of parts of the second object is formed of a bone formed of either of a point and a line segment, and

arranging an object further includes calculating a normal vector extending from a point on the bone, at which a distance between the designated three-dimensional position and the bone is shortest, toward the designated three-dimensional position, and determining the orientation of the designating object, according to the normal vector.

9. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 7, wherein

the display control program further causes said one or more processors to function and perform operations comprising:

setting an angle of rotation of the designating object according to the designated three-dimensional position relative to the designated part, and

rotating the designating object about the normal line by the angle of rotation to arrange the designating object in the virtual space.

10. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 5, wherein

the display control displays the designating object in different modes, depending on whether or not the designated position exists within the area in which the second object is displayed.

11. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 5, wherein

in the case where the designated position exists within the area in which the second object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and the depth value of the image at the designated position, and

in the case where the designated position exists outside of the area in which the second object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and a constant depth value.

12. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

the display area is displayable in a stereoscopically visible manner, and

the display control program causes the display control to perform operations comprising:

generating an image for a left eye and an image for a right eye by taking images of the virtual space by a virtual stereo camera,

acquiring either of a depth value at the designated position on the image for the left eye or the image for the right eye, and a depth value at the designated position on a third image acquired by taking an image of the virtual space by a third virtual camera having the same imaging direction as that of the virtual stereo camera, and

stereoscopically displaying, on the display area, an image taken of the virtual space including the object arranged therein by the object arranging means.

13. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 12, wherein

the third virtual camera is set between a virtual camera at the left and a virtual camera at the right which are components of the virtual stereo camera, and

the display control program causes the display control to perform operations of:

acquiring the depth value of the third image at the designated position on the third image, and

calculating the designated three-dimensional position based on the designated position on the third image and the depth value of the third image.

14. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 13, wherein

the third virtual camera is set at a middle location between the virtual camera at the left and the virtual camera at the right which are the components of the virtual stereo camera.

15. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 12, wherein

a position designator is provided via a second display area, which is different from the first display area, and which detects a touch position on the second display area, and

the display control program further causes said one or more processors to function and perform operations comprising:

generating a third image which is acquired by taking an image of the virtual space by a third virtual camera having the same imaging direction as that of the virtual stereo camera,

acquiring a depth value on the third image at a designated position corresponding to the touch position,

and wherein the display control displays, on the display area, the image for the left eye and the image for the right eye, and displays, on the second display area, the third image.

16. A display control apparatus comprising:

image generator that generates a two-dimensional image, in which depth values are set in association with each pixel position of the two-dimensional image, by taking an image of a three-dimensional virtual space by a virtual camera;

depth value acquirer that acquires a depth value associated to pixel position a position designated by a user on the two-dimensional image via an input touch position designation;

one or more processors configured to calculate, based on an acquired depth value, a spatial position in a depth direction in the virtual space as viewed by the virtual camera;

designating object generator that produces a virtual designating object in the virtual space at the calculated spatial position in the depth direction; and

display controller that displays on a first display area a first image taken of the virtual space which includes the virtual designating object produced therein and which displays a second image on a second display area, the second image being based upon the first image, wherein the input touch position designation is provided via a second display area, which is different from the first display area and which is capable of detecting a touch position on the second display area, and wherein the depth value is acquired based upon the touch position and a corresponding pixel position in the second image displayed on the second display area.

17. A display control system comprising:

a processing system, including at least one computer processor, the processing system being configured to;

generate a two-dimensional image, in which depth values are set, by taking an image of a three-dimensional virtual space by a virtual camera and setting depth values that are associated with each pixel position of the two-dimensional image;

acquire an associated depth value for a pixel of the image corresponding to a designated position on the image, wherein the designated position on the image is provided by a user via an input device touch position designation;

calculate, using said at least one processor, a spatial position in a depth direction in the virtual space as viewed by the virtual camera based on a depth value associated to the designated position on the image;

produce a virtual designating object at the calculated spatial position in the depth direction in the virtual space; and

display, on a first display area, a first image taken of the virtual space which includes the virtual designating object produced therein and display a second image on a second display area, the second image being based upon the first image, wherein the input touch position designation is provided via a second display area, which is different from the first display area and which is capable of detecting a touch position on the second display area, and wherein the depth value is acquired based upon the touch position and a corresponding pixel position in the second image displayed on the second display area.

18. A display control method implemented using an information processing apparatus having one or more processors for controlling a displaying of an image on a display device, the method comprising:

generating a two-dimensional image, in which depth values are set, by taking a two-dimensional image of a three-dimensional virtual space by a virtual camera and setting depth values which are associated with each pixel position of the image;

acquiring the associated depth value for a pixel of the image at a designated position on the image, wherein the designated position on the image is provided by a user via an input device touch position designation;

calculating, using said one or more processors, based on a depth value associated to the designated position on the image, a spatial position in a depth direction in the virtual space as viewed by the virtual camera;

producing a virtual designating object at the calculated spatial position in the depth direction in the virtual space; and

displaying, on a first display area, a first image taken of the virtual space which includes the virtual designating object produced therein and displaying a second image on a second display area, the second image being based upon the first image, wherein the input touch position designation is provided via a second display area, which is different from the first display area and which is capable of detecting a touch position on the second display area, and wherein the depth value is acquired based upon the touch position and a associated pixel position in the second image displayed on the second display area.

19. A non-transitory computer-readable storage medium having stored therein a display control program having computer-executable code or instructions which, when executed by one or more processors of a computerized display control, causes the display control to function and perform operations comprising:

generating an image in which depth values are set, by taking a two-dimensional image of a three-dimensional virtual space by a virtual camera and setting depth values to be associated with each pixel position of the image;

acquiring an associated depth value for a pixel of the image corresponding to a designated position on the image, wherein the designated position on the image is provided via an input device;

calculating, based on the depth value of the designated position on the image, a spatial position in a depth direction in the virtual space as viewed from the virtual camera, wherein the calculated spatial position corresponds to a designated three-dimensional position in the virtual space, based on both the designated position on the image and the depth value of the image at the designated position;

producing the virtual designating object at the designated three-dimensional position in the virtual space;

producing a second virtual object to be displayed in the virtual space, wherein the second object is formed of a plurality of parts;

generating an image by taking, via the virtual camera, an image of the virtual space including the second object and, if the designated position exists in an area in which the second object is displayed, determining, based on the designated three-dimensional position, an orientation of the designating object;

determining, based on the designated position, a designated part from among the plurality of parts of the second virtual object;

determining the orientation of the designating object according to the designated part and arranging the designating object in the virtual space in the determined orientation; and

displaying, on a first display area, an image taken of the virtual space which includes the virtual designating object arranged therein, wherein the designating object is a cursor object indicating a position within the virtual space.

20. A non-transitory computer-readable storage medium having stored therein a display control program having computer-executable code or instructions which, when executed by one or more processors of a computerized display control, causes the display control to function and perform operations comprising:

generating an image in which depth values are set, by taking a two-dimensional image of a three-dimensional virtual space by a virtual camera and setting depth values to be associated with each pixel position of the image;

acquiring an associated depth value for a pixel of the image corresponding to a designated position on the image, wherein the designated position on the image is provided via an input device;

calculating, based on the depth value of the designated position on the image, a spatial position in a depth direction in the virtual space as viewed from the virtual camera, wherein the calculated spatial position corresponds to a designated three-dimensional position in the virtual space, based on both the designated position on the image and the depth value of the image at the designated position;

producing the virtual designating object at the designated three-dimensional position in the virtual space;

producing a second virtual object to be displayed in the virtual space;

generating an image by taking, via the virtual camera, an image of the virtual space including the second object; and

if the designated position exists in an area in which the second object is displayed, determining, based on the designated three-dimensional position, an orientation of the designating object and arranging in the virtual space the designating object in the determined orientation; and

if the designated position exists within the area in which the second object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and the depth value of the image at the designated position; and

if the designated position exists outside of the area in which the second object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and a constant depth value; and

displaying, on a first display area, an image taken of the virtual space which includes the virtual designating object arranged therein, wherein the designating object is a cursor object indicating a position within the virtual space.