Computer-readable storage medium having stored therein display control program, display control apparatus, display control system, and display control method

• US 10,506,218 B2
• Filed: 12/07/2016
• Issued: 12/10/2019
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Oyagi, Yasuyuki, Morii, Junji, Matoba, Taku, Sato, Katsuhisa
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Imperial, Jed-Justin
• Application Number:
  US15/372,148
• Publication Number:
  US 20170094264A1
• Time in Patent Office: 1,098 Days
• Field of Search: None
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: G02F 1/13338 Input devices, e.g. touch panels
• CPC Class Codes: G06F 1/1692 the I/O peripheral being a secondary touch screen used as control interface, e.g. virtual buttons or sliders
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

An image taken of a virtual space including a user-operable child object and user-inoperable furniture object is stereoscopically displayed on a screen of a stereoscopic image display device. On the other hand, an image taken, of the same virtual space, from the same direction is displayed in a planar manner on a screen of a planar image display device. On the screen of the planar image display device, a child object image, in which a silhouette of the child object displayed on the stereoscopic image display device is displayed. A user touches on the screen of the planar image display device, while seeing the child object displayed on the stereoscopic image display device, thereby operating the child object.

Claims

1. A non-transitory computer-readable storage medium having stored therein a display control program having computer-executable code or instructions which, when executed by one or more processors of a computerized display control, causes the display control to function and perform operations comprising:

generating an image in which depth values are set by taking a two-dimensional image of a three-dimensional virtual space taken by a virtual camera and setting depth values to be associated with each pixel position of the generated image, an image taken of the three-dimensional virtual space including an object, where the object is to be displayed in the virtual space;

acquiring an associated depth value for a pixel of the generated image corresponding to a designated position on the generated image, wherein the designated position on the generated image is provided via an input device;

calculating, based on the depth value of the designated position on the generated image, a spatial position in a depth direction in the virtual space as viewed from the virtual camera;

producing a virtual designating object at the calculated spatial position in the depth direction in the virtual space; and

displaying, on a first display area, the image taken of the three-dimensional virtual space which includes the virtual designating object and the object arranged therein,

in the case where the designated position exists in an area in which the object is displayed, the virtual designating object is displayed in a three-dimensional manner, the virtual designating object having an orientation, and in the case where the designated position does not exist in the area in which the object is displayed, the virtual designating object is displayed in a planar manner.

2. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 1, wherein

the calculated spatial position corresponds to a designated three-dimensional position in the virtual space, based on both the designated position on the generated image and the depth value of the generated image at the designated position, and wherein the display control program causes the display control to perform operations comprising:

producing the virtual designating object at the designated three-dimensional position in the virtual space.

3. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

the designated three-dimensional position in the virtual space is calculated based on a location of the designated position on the two-dimensional image in up-down and left-right directions in the virtual space as viewed from the virtual camera and the depth value associated with the designated position.

4. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

before the virtual designating object is produced, the virtual designating object does not exist at the calculated spatial position, and in the case where the virtual designating object is produced, the virtual designating object appears at the calculated spatial position.

5. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein the display control program causes the display control to perform operations comprising:

arranging the object to be displayed in the virtual space,

generating the image of the three-dimensional virtual space by taking, via the virtual camera, an image of the three-dimensional virtual space including the object, and

in the case where the designated position exists in the area in which the object is displayed, determining, based on the designated three-dimensional position, the orientation of the virtual designating object, and arranging in the virtual space the virtual designating object in the determined orientation.

6. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 5, wherein

the orientation of the virtual designating object is set along a surface of the object.

7. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

in the case where the designated position exists within the area in which the object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and the depth value of the generated image at the designated position, and

in the case where the designated position exists outside of the area in which the object is displayed, calculating the designated three-dimensional position in the virtual space based on the designated position and a constant depth value.

8. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

the first display area is displayable in a stereoscopically visible manner, and

the display control program causes the display control to perform operations comprising:

generating an image for a left eye and an image for a right eye by taking images of the virtual space by a virtual stereo camera,

acquiring either of a depth value at the designated position on the image for the left eye or the image for the right eye, and the depth value at the designated position on the generated image, where the virtual camera has the same imaging direction as that of the virtual stereo camera, and

stereoscopically displaying, on the first display area, the image taken of the three-dimensional virtual space which is formed by the image for the left eye and the image for the right eye including the virtual designating object arranged therein.

9. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 8, wherein

the virtual camera is set between a virtual camera at the left and a virtual camera at the right which are components of the virtual stereo camera.

10. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 9, wherein

the virtual camera is set at a middle location between the virtual camera at the left and the virtual camera at the right which are the components of the virtual stereo camera.

11. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 8, wherein

a position designator is provided via a second display area, which is different from the first display area, and which detects a touch position on the second display area, and

wherein the depth value on the generated image at the designated position corresponding to the touch position,

and wherein the display control displays, on the first display area, the image for the left eye and the image for the right eye, and displays, on the second display area, the generated image.

12. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 2, wherein

a position designator is provided via a second display area, which is different from the first display area, and which detects a touch position on the second display area, and

the display control program causes the display control to perform operations comprising:

acquiring the depth value on the generated image displayed on the second display area at the designated position corresponding to the touch position, and

displaying, on the first display area, the image taken of the three-dimensional virtual space including the virtual designating object arranged therein, and displaying, on the second display area, the generated image taken of the virtual space.

13. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 1, wherein

in the case where the designated position does not exist in the area in which the object is displayed, the virtual designating object is arranged in a predetermined orientation.

14. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 1, wherein the image taken of the virtual space comprising at least one selectable object and at least one non-selectable object,

wherein the display control program causes the display control to perform operations comprising:

displaying the generated image in which depth values are set by taking the two-dimensional image of the three-dimensional virtual space by the virtual camera, wherein the generated image comprises the at least one selectable object without the at least one non-selectable object.

15. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 14, in the case where a designated position of the virtual designating object exists in an area of the at least one selectable object, the at least one selectable object in the virtual space changes in accordance with an operation on the at least one selectable object in the generated image.

16. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 1, wherein the display control program causes the display control to perform operations comprising:

displaying the generated image in which depth values are set by taking the two-dimensional image of the three-dimensional virtual space by the virtual camera, wherein an imaging direction of the generated image is the same as an imaging direction of the image taken of the virtual space.

17. A non-transitory computer-readable storage medium having stored therein a display control program having computer-executable code or instructions which, when executed by one or more processors of a computerized display control, causes the display control to function and perform operations comprising:

generating an image in which depth values are set by taking a two-dimensional image of a three-dimensional virtual space taken by a virtual camera and setting depth values to be associated with each pixel position of the generated image;

acquiring an associated depth value for a pixel of the generated image corresponding to a designated position on the generated image, wherein the designated position on the generated image is provided via an input device;

calculating, based on the depth value of the designated position on the generated image, a spatial position in a depth direction in the virtual space as viewed from the virtual camera, the calculated spatial position corresponds to a designated three-dimensional position in the virtual space, based on both the designated position on the generated image and the depth value of the generated image at the designated position;

producing a virtual designating object at the calculated spatial position in the depth direction in the virtual space which corresponds to the designated three-dimensional position in the virtual space;

arranging an object to be displayed in the virtual space;

generating an image of the three-dimensional virtual space by taking, via the virtual camera, an image of the three-dimensional virtual space including the object;

displaying, on a first display area, the generated image taken of the of three-dimensional virtual space which includes the virtual designating object and the object arranged therein,

in the case where the designated position exists in an area in which the object is displayed, determining, based on the designated three-dimensional position, an orientation of the virtual designating object, and arranging in the virtual space the virtual designating object in the determined orientation, and wherein

the object is formed of a plurality of parts,

the display control program further causes one or more processors of the display control to function and perform operations comprising;

determining, based on the designated position, a designated part from among the plurality of parts of the object, and

determining the orientation of the virtual designating object, according to the designated part, and arranges in the virtual space the virtual designating object in the determined orientation.

18. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 17, wherein

arranging the virtual designating object further includes determining the orientation of the virtual designating object, according to a normal line of the designated part at the designated three-dimensional position.

19. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 18, wherein

each of the plurality of parts of the object is formed of a bone formed of either of a point and a line segment, and

arranging the virtual designating object further includes calculating a normal vector extending from a point on the bone, at which a distance between the designated three-dimensional position and the bone is shortest, toward the designated three-dimensional position, and determining the orientation of the virtual designating object, according to the normal vector.

20. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 18, wherein

the display control program further causes said one or more processors to function and perform operations comprising:

setting an angle of rotation of the virtual designating object according to the designated three-dimensional position relative to the designated part, and

rotating the virtual designating object about the normal line by the angle of rotation to arrange the virtual designating object in the virtual space.

21. The non-transitory computer-readable storage medium having stored therein the display control program according to claim 17, wherein

the display control displays the virtual designating object in different modes, depending on whether or not the designated position exists within the area in which the object is displayed.

22. A display control apparatus comprising:

image generator that generates a two-dimensional image, in which depth values are set, by taking an image of a three-dimensional virtual space by a virtual camera, the image taken of the three-dimensional virtual space including an object, where the object is to be displayed in the virtual space;

depth value acquirer that acquires a depth value for a pixel of the generated image at a designated position designated on the generated image;

one or more processors configured to calculate, based on the depth value of the designated position on the generated image, a spatial position in a depth direction in the virtual space as viewed by the virtual camera;

designating object generator that produces a virtual designating object at the calculated spatial position in the depth direction; and

display controller that displays on a display area the image taken of the three-dimensional virtual space which includes the virtual designating object and the object,

in the case where the designated position exists in an area in which the object is displayed, the virtual designating object is displayed in a three-dimensional manner, the virtual designating object having an orientation, and in the case where the designated position does not exist in the area in which the object is displayed, the virtual designating object is displayed in a planar manner.

23. A display control system comprising:

a processing system, including at least one computer processor, the processing system being configured to;

generate a two-dimensional image, in which depth values are set, by taking an image of a three-dimensional virtual space by a virtual camera, the image of the three-dimensional virtual space including an object, where the object is to be displayed in the virtual space;

acquire a depth value for a pixel of the generated image at a designated position on the generated image;

calculate, using said at least one processor, a spatial position in a depth direction in the virtual space as viewed by the virtual camera based on the depth value of the designated position on the generated image;

produce a virtual designating object at the calculated spatial position in the depth direction; and

display on a display area the image taken of the three-dimensional virtual space which includes the virtual designating object and the object,

in the case where the designated position exists in an area in which the object is displayed, the virtual designating object is displayed in a three-dimensional manner, the virtual designating object having an orientation, and in the case where the designated position does not exist in the area in which the object is displayed, the virtual designating object is displayed in a planar manner.

24. A display control method implemented using an information processing apparatus having one or more processors for controlling a displaying of an image on a display device, the method comprising:

generating a two-dimensional image, in which depth values are set, by taking an image of a three-dimensional virtual space by a virtual camera, the image of the three-dimensional virtual space includes an object, where the object is to be displayed in the virtual space;

acquiring the depth value for a pixel of the generated image at a designated position on the generated image;

calculating, using said one or more processors, said calculating based on the depth value of the designated position on the generated image, a spatial position in a depth direction in the virtual space as viewed by the virtual camera;

producing a virtual designating object at the calculated spatial position in the depth direction; and

displaying, on a display area, the image taken of the three-dimensional virtual space which includes the virtual designating object and the object,

in the case where the designated position exists in an area in which the object is displayed, the virtual designating object is displayed in a three-dimensional manner, the virtual designating object having an orientation, and in the case where the designated position does not exist in the area in which the object is displayed, the virtual designating object is displayed in a planar manner.