INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, INFORMATION PROCESSING DEVICE AND TANGIBLE RECORDING MEDIUM RECORDING INFORMATION PROCESSING PROGRAM

• US 20120218298A1
• Filed: 02/15/2012
• Published: 08/30/2012
• Est. Priority Date: 02/25/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: HAYAKAWA, Takeshi
• Granted Patent: US 9,064,335 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An example system includes an information processing device, which includes: an arrangement item including a first marker; an arrangement region providing object which provides an arrangement region and includes a plurality of second markers; and an information processing device including: an imaging processing unit to generate a captured image with an imaging device; a positional relationship judgment unit to judge a positional relationship between the first marker of the arrangement item and at least one of the second markers, from a captured image which includes the first marker of the arrangement item and at least one of the plurality of second markers of the arrangement region providing object; an information superimposition unit to superimpose predetermined information based on the positional relationship, onto the captured image; and a display processing unit to cause a display device to display the captured image on which the predetermined information is superimposed.

Claims

1. An information processing system, comprising:

an arrangement item including a first marker;

an arrangement region providing object which provides an arrangement region for the arrangement item and includes a plurality of second markers; and

an information processing device including;

an imaging processing unit to generate a captured image by capturing images with an imaging device;

a positional relationship judgment unit to judge a positional relationship between the first marker of the arrangement item and at least one of the second markers, from a captured image generated by the imaging processing unit which includes the first marker of the arrangement item and at least one of the plurality of second markers of the arrangement region providing object;

an information superimposition unit to superimpose predetermined information based on the positional relationship, onto the captured image generated by the imaging processing unit; and

a display processing unit to cause a display device to display the captured image on which the predetermined information is superimposed.

2. The information processing system according to claim 1, wherein

the first marker and the plurality of second markers respectively have a shape which enables detection of a position of the marker and an attitude of the marker at the position;

and

the positional relationship judgment unit judges one or a combination of a distance between the first marker and at least one of the second markers, an orientation of the first marker with respect to at least one of the second markers, and a direction of the position of the first marker with respect to at least one of the second markers, as a positional relationship between the first marker and at least one of the second markers, based on the shape of the first marker and the shape of at least one of the second markers.

3. The information processing system according to claim 1, wherein the information superimposition unit superimposes the predetermined information at a position on the arrangement region providing object on the captured image which is based on the positional relationship judged by the positional relationship judgment unit.

4. The information processing system according to claim 1, wherein

arrangement of the arrangement item on the arrangement region providing object is allowed to be changed by a user;

the imaging processing unit generates a captured image by capturing images successively at predetermined intervals by the imaging device;

the positional relationship judgment unit detects change in the positional relationship of the first marker and at least one second marker due to the change in the arrangement of the arrangement item, from the captured images generated at predetermined intervals; and

the information superimposition unit superimposes the predetermined information in accordance with the change in the positional relationship.

5. The information processing system according to claim 4, wherein the information superimposition unit superimposes the predetermined information, when the change is detected in the distance between the first marker and at least one second marker.

6. The information processing system according to claim 4, wherein the information superimposition unit superimposes information indicating a passing of time from previously superimposition of information, as the predetermined information, when the change in a distance between the first marker and at least one second marker is detected as change in the positional relationship.

7. The information processing system according to claim 4, wherein the information superimposition unit superimposes the predetermined information when change in an orientation of the first marker with respect to the at least one second marker is detected as change in the positional relationship.

8. The information processing system according to claim 4, wherein

the at least one second marker defines a position and direction on the arrangement region with reference to the marker, by means of the shape of the marker;

and

the information superimposition unit superimposes the predetermined information when change in a direction, in which the first marker is positioned with reference to the at least one second marker, is detected as change in the positional relationship.

9. The information processing system according to claim 1, wherein

the arrangement region providing object includes a plurality of divided regions of a predetermined size in the arrangement region;

the plurality of second markers are respectively allocated to a portion of the divided regions; and

the information superimposition unit superimposes the predetermined information based on a divided region to which the at least one second marker is allocated.

10. The information processing system according to claim 9, wherein the information superimposition unit superimposes the predetermined information, when the positional relationship judgment unit judges that the first marker is arranged in the divided region allocated to the at least one second marker.

11. The information processing system according to claim 9, wherein the information superimposition unit superimposes the predetermined information, when the positional relationship judgment unit judges that the first marker is arranged in a range including the predetermined number of divided regions about a periphery of the divided region allocated to the at least one second marker.

12. The information processing system according to claim 9, wherein the information superimposition unit superimposes the predetermined information, when the first marker is removed from a range including the predetermined number of divided regions about a periphery of the divided region allocated to the at least one second marker.

13. The information processing system according to claim 9, wherein the information superimposition unit superimposes information which induces passing of time of the information superimposed on the captured image, as the predetermined information, when change in the divided region where the first marker is positioned is detected by the positional relationship judgment unit.

14. The information processing system according to claim 9, further comprising a divided region judgment unit to judge a positional relationship between the first marker and the second marker based on the arrangement of the plurality of divided regions, from the positional relationship judged by the positional relationship judgment unit.

15. The information processing system according to claim 1, wherein, when the captured image generated by the imaging processing unit includes the first marker and a plurality of the second markers, the information superimposition unit superimposes the predetermined information in accordance with a positional relationship between the first marker and a second marker located closest to the first marker.

16. The information processing system according to claim 1, wherein

the first marker and the plurality of second markers are respective indicators of display positions of corresponding objects;

and

the display processing unit causes a display device to display respectively corresponding objects after implementing superimposition at positions corresponding to the first marker and the plurality of second markers which are included in the captured image generated by the imaging processing unit.

17. The information processing system according to claim 1, wherein

the arrangement item is a card;

the arrangement region providing object is a board; and

the information processing device further includes a game development control unit to control development of a game in which a status changes due to the card being arranged on the board by a user.

18. An information processing method for an information processing system comprising:

an arrangement item including a first marker;

an arrangement region providing object which provides an arrangement region for the arrangement item and includes a plurality of second markers; and

an information processing device which is connected to an imaging device, wherein

the information processing device executes;

generating a captured image by capturing images with an imaging device;

judging a positional relationship between the first marker and at least one second marker, from the generated captured image, the image being configured to include the first marker of the arrangement item and at least one of the plurality of second markers of the arrangement region providing object;

superimposing predetermined information based on the positional relationship, onto the generated captured image; and

causing a display device to display a captured image on which the predetermined information is superimposed.

19. An information processing apparatus, comprising:

an imaging processing unit to generate a captured image by capturing images with an imaging device;

a positional relationship judgment unit to judge a positional relationship between a first marker provided on an arrangement item and at least one of a plurality of second markers provided on an arrangement region providing object which provides an arrangement region to the arrangement item, from a captured image generated by the imaging processing unit which includes the first marker and at least one second marker;

an information superimposition unit to superimpose predetermined information based on the positional relationship, onto a captured image generated by the imaging processing unit; and

a display processing unit to cause a display device to display a captured image on which the predetermined information is superimposed.

20. A tangible recording medium recording an information processing program executed by a computer of an information processing device which carries out imaging by an imaging device,

the program causing the computer to function as:

an imaging processing unit to generate a captured image by capturing images with an imaging device;

a positional relationship judgment unit to judge a positional relationship between a first marker provided on an arrangement item and at least one of a plurality of second markers provided on an arrangement region providing object which provides an arrangement region to the arrangement item, from a captured image generated by the imaging processing unit which includes the first marker and the at least one second marker;

an information superimposition unit to superimpose predetermined information based on the positional relationship, onto a captured image generated by the imaging processing unit; and

a display processing unit to cause a display device to display a captured image on which the predetermined information is superimposed.