System, method, device and computer-readable medium recording information processing program for superimposing information

• US 9,064,335 B2
• Filed: 02/15/2012
• Issued: 06/23/2015
• Est. Priority Date: 02/25/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Hayakawa, Takeshi
• Primary Examiner(s): Richer, Aaron M
• Assistant Examiner(s): NGUYEN, ANH TUAN V
• Application Number:
  US13/396,937
• Publication Number:
  US 20120218298A1
• Time in Patent Office: 1,224 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An example system includes an information processing device, which includes: an arrangement item including a first marker; an arrangement region providing object which provides an arrangement region and includes a plurality of second markers; and an information processing device including: an imaging processing unit to generate a captured image with an imaging device; a positional relationship judgment unit to judge a positional relationship between the first marker of the arrangement item and at least one of the second markers, from a captured image which includes the first marker of the arrangement item and at least one of the plurality of second markers of the arrangement region providing object; an information superimposition unit to superimpose predetermined information based on the positional relationship, onto the captured image; and a display processing unit to cause a display device to display the captured image on which the predetermined information is superimposed.

Claims

1. An information processing system, comprising:

an arrangement item including a first marker, wherein the arrangement item is a physical item;

an arrangement region providing object which provides an arrangement region for the arrangement item and which includes a plurality of second markers, wherein the arrangement region providing object is a physical object, and wherein arrangement of the arrangement item on the arrangement region providing object is changeable by a user; and

an information processing device comprising a processing system including at least one processor, the processing system configured to perform operations including;

capturing images successively at predetermined intervals by controlling an imaging device;

determining a positional relationship between the first marker of the arrangement item and at least one of the plurality of second markers from the captured images which include the first marker of the arrangement item and the at least one of the plurality of second markers of the arrangement region providing object for a predetermined period of time, wherein the positional relationship includes an orientation of the first marker with respect to the at least one of the plurality of second markers;

buffering the determined positional relationship for the predetermined period of time;

monitoring the buffered positional relationship for the predetermined period of time;

detecting, based upon the monitoring, a change in the positional relationship of the first marker and the at least one of the plurality of second markers due to one or more changes in the arrangement of the arrangement item, wherein the change in the positional relationship includes a change in the orientation of the first marker with respect to the at least one of the plurality of second markers;

superimposing predetermined information, in accordance with the detected change in the positional relationship of the first marker with respect to the at least one of the plurality of second markers, onto a captured image, wherein the predetermined information depicts one or more virtual actions relating to the first marker or the at least one of the plurality of second markers; and

causing a display device to display the captured image on which the predetermined information is superimposed.

2. The information processing system according to claim 1, wherein

the first marker and the plurality of second markers respectively have a shape which enables detection of a position of the first marker or any of the plurality of second markers and an attitude of the first marker or any of the plurality of the second markers at the position;

and

wherein the operations further comprise determining one or a combination of a distance between the first marker and at least one of the second markers, an orientation of the first marker with respect to at least one of the second markers, and a direction of the position of the first marker with respect to at least one of the second markers, as the positional relationship between the first marker and at least one of the second markers, based on the shape of the first marker and the shape of at least one of the second markers.

3. The information processing system according to claim 1, wherein the operations further comprise superimposing the predetermined information at a position on the arrangement region providing object on the captured image which is based on the determined positional relationship.

4. The information processing system according to claim 1, wherein the operations further comprise superimposing the predetermined information when the detected change includes a change in the distance between the first marker and at least one second marker.

5. The information processing system according to claim 1, wherein the operations further comprise superimposing information indicating a passing of time from a previous superimposition of information, as the predetermined information, when the detected change includes a change in a distance between the first marker and at least one second marker.

6. The information processing system according to claim 1, wherein

the arrangement region providing object includes a plurality of divided regions of a predetermined size in the arrangement region, wherein

the plurality of second markers are respectively allocated to a portion of the divided regions, and wherein

the operations further comprise superimposing the predetermined information based on a divided region to which the at least one second marker is allocated.

7. The information processing system according to claim 6, wherein the operations further comprise superimposing the predetermined information, when the determining a positional relationship includes determining that the first marker is arranged in the divided region allocated to the at least one second marker.

8. The information processing system according to claim 6, wherein the operations further comprise superimposing the predetermined information, when the determining a positional relationship includes determining that the first marker is arranged in a range including the predetermined number of divided regions about a periphery of the divided region allocated to the at least one second marker.

9. The information processing system according to claim 6, wherein the operations further comprise superimposing the predetermined information when the first marker is removed from a range including the predetermined number of divided regions about a periphery of the divided region allocated to the at least one second marker.

10. The information processing system according to claim 6, wherein the operations further comprise superimposing information which indicates passing of time from a previous superimposition of information on the captured image, as the predetermined information, when the detected change includes a change in the divided region where the first marker is positioned.

11. The information processing system according to claim 6, further comprising operations including determining a positional relationship between the first marker and the at least one second marker based on the arrangement of the plurality of divided regions.

12. The information processing system according to claim 1, wherein, when the generated captured image includes the first marker and a plurality of the second markers, and wherein the operations further comprise superimposing the predetermined information in accordance with a positional relationship between the first marker and a second marker located closest to the first marker.

13. The information processing system according to claim 1, wherein

the first marker and the plurality of second markers are respective indicators of display positions of corresponding objects, and

wherein the operations further comprise causing a display device to display respectively corresponding objects after implementing superimposition at positions corresponding to the first marker and the plurality of second markers which are included in the generated captured image.

14. The information processing system according to claim 1, wherein

the arrangement item is a card;

and

the arrangement region providing object is a board and

wherein the operations further comprise controlling development of a game in which a status changes due to the card being arranged on the board by a user.

15. An information processing system, comprising:

an arrangement item including a first marker, wherein the arrangement item is a physical item;

an arrangement region providing object which provides an arrangement region for the arrangement item and which includes a plurality of second markers, wherein the arrangement region providing object is a physical object, and wherein arrangement of the arrangement item on the arrangement region providing object is changeable by a user; and

an information processing device comprising a processing system including at least one processor, the processing system configured to perform operations including;

capturing images successively at predetermined intervals by controlling an imaging device;

determining a positional relationship between the first marker of the arrangement item and at least one of the plurality of second markers from the captured images which include the first marker of the arrangement item and the at least one of the plurality of second markers of the arrangement region providing object for a predetermined period of time, wherein the positional relationship includes an orientation of the first marker with respect to the at least one of the plurality of second markers;

buffering the determined positional relationship for the predetermined period of time;

monitoring the buffered positional relationship for the predetermined period of time;

detecting, based upon the monitoring, a change in the positional relationship of the first marker and the at least one of the plurality of second markers due to one or more changes in the arrangement of the arrangement item, wherein the change in the positional relationship includes a change in the orientation of the first marker with respect to the at least one of the plurality of second markers;

superimposing predetermined information, in accordance with the detected change in the positional relationship of the first marker with respect to the at least one of the plurality of second markers, onto a captured image, wherein the predetermined information depicts one or more virtual actions relating to the first marker or the at least one of the plurality of second markers; and

causing a display device to display the captured image on which the predetermined information is superimposed,

wherein

the at least one second marker defines a position and direction on the arrangement region with reference to the first marker, by means of the shape of the first marker; and

the operations further comprise superimposing the predetermined information when the detected change includes a change in a direction in which the first marker is positioned with reference to the at least one second marker.

16. An information processing method for an information processing system comprising:

an arrangement item including a first marker, wherein the arrangement item is a physical item;

an arrangement region providing object which provides an arrangement region for the arrangement item and which includes a plurality of second markers, wherein the arrangement region providing object is a physical object, and wherein arrangement of the arrangement item on the arrangement region providing object is changeable by a user; and

an information processing device which is communicably connected to an imaging device, wherein the information processing device executes operations comprising;

capturing images successively at predetermined intervals by controlling the imaging device;

determining a positional relationship between the first marker and at least one of the plurality of second markers from the captured images which include the first marker of the arrangement item and the at least one of the plurality of second markers of the arrangement region providing object for a predetermined period of time, wherein the positional relationship includes an orientation of the first marker with respect to the at least one of the plurality of second markers;

buffering the determined positional relationship for the predetermined period of time;

monitoring the buffered positional relationship for the predetermined period of time;

detecting, based upon the monitoring, a change in the positional relationship of the first marker and the at least one of the plurality of second markers due to one or more changes in the arrangement of the arrangement item, wherein the change in the positional relationship includes a change in the orientation of the first marker with respect to the at least one of the plurality of second markers;

superimposing predetermined information, in accordance with the detected change in the positional relationship of the first marker with respect to the at least one of the plurality of second markers, onto a captured image, wherein the predetermined information depicts one or more virtual actions relating to the first marker or the at least one of the plurality of second markers; and

causing a display device to display a captured image on which the predetermined information is superimposed.

17. An information processing apparatus, comprising one or more processors configured to:

capture images successively at predetermined intervals by controlling an imaging device;

determine a positional relationship between a first marker provided on an arrangement item and at least one of a plurality of second markers provided on an arrangement region providing object which provides an arrangement region to the arrangement item, from the captured images which include the first marker and at least one of the plurality of second markers for a predetermined period of time, wherein the determined positional relationship includes an orientation of the first marker with respect to the at least one of the plurality of second markers, wherein the arrangement item is a physical item and the arrangement region providing object is a physical object, and wherein arrangement of the arrangement item on the arrangement region providing object is changeable by a user;

buffer the determined positional relationship for the predetermined period of time;

monitor the buffered positional relationship for the predetermined period of time; and

detect, based upon the monitoring, a change in the positional relationship of the first marker and the at least one of the plurality of second markers due to one or more changes in the arrangement of the arrangement item, wherein the change in the positional relationship includes a change in the orientation of the first marker with respect to the at least one of the plurality of second markers;

superimpose predetermined information, in accordance with the detected change in the positional relationship of the first marker with respect to the at least one of the plurality of second markers, onto a captured image, wherein the predetermined information depicts one or more virtual actions relating to the first marker or the at least one of the plurality of second markers; and

cause a display device to display the at least one captured image on which the predetermined information is superimposed.

18. A non-transitory computer-readable storage medium storing an information processing program, which, when executed by at least one computer of an information processing device, causes the at least one computer to perform operations comprising:

capturing images successively at predetermined intervals by controlling an imaging device;

determining a positional relationship between a first marker provided on an arrangement item and at least one of a plurality of second markers provided on an arrangement region providing object which provides an arrangement region to the arrangement item, from the captured images which include the first marker and the at least one of the plurality of second markers for a predetermined period of time, wherein the determined positional relationship includes an orientation of the first marker with respect to the at least one of the plurality of second markers, wherein the arrangement item is a physical item and the arrangement region providing object is a physical object, and wherein arrangement of the arrangement item on the arrangement region providing object is changeable by a user;

buffering the determined positional relationship for the predetermined period of time;

monitoring the buffered positional relationship for the predetermined period of time; and

detecting, based upon the monitoring, a change in the positional relationship of the first marker and the at least one of the plurality of second markers due to one or more changes in the arrangement of the arrangement item, wherein the change in the positional relationship includes a change in the orientation of the first marker with respect to the at least one of the plurality of second markers;

superimposing predetermined information, in accordance with the detected change in the positional relationship of the first marker with respect to the at least one of the plurality of second markers, onto a captured image, wherein the predetermined information depicts one or more virtual actions relating to the first marker or the at least one of the plurality of second markers; and

causing a display device to display a captured image on which the predetermined information is superimposed.