CLIENT DEVICE

• US 20130257907A1
• Filed: 01/31/2013
• Published: 10/03/2013
• Est. Priority Date: 03/30/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Mobile Communications Incorporated (Sony Group Corp.)
• Inventors: MATSUI, Daisuke
• Granted Patent: US 9,293,118 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality

Abstract

An information processing apparatus comprising an image recognition unit that detects whether an image preset as a marker image exists in a captured image; a spatial recognition unit that, when the marker image is detected by the image recognition unit, constructs a real-world coordinate system computed from a position and orientation of the marker image, and computes information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system; and a virtual object display unit that acquires at least one of the data stored in the storage unit, data existing on a network, and data transmitted from the one or more other information processing apparatuses and received by the communication interface, places the acquired data in the constructed real-world coordinate system, and controls the display unit to display the acquired data as one or more virtual objects.

Claims

1. An information processing apparatus comprising:

a communication interface that communicates with one or more other information processing apparatuses;

an imaging unit that captures an image corresponding to a subject and outputs the captured image;

a display unit that displays the captured image;

an image recognition unit that detects whether an image preset as a marker image exists in the captured image;

a storage unit storing data;

a spatial recognition unit that, when the marker image is detected by the image recognition unit, constructs a real-world coordinate system computed from a position and orientation of the marker image, and computes information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system;

a virtual object display unit that acquires at least one of the data stored in the storage unit, data existing on a network, and data transmitted from the one or more other information processing apparatuses and received by the communication interface, places the acquired data in the constructed real-world coordinate system, and controls the display unit to display the acquired data as one or more virtual objects; and

a controller that controls the communication interface to transmit information corresponding to the position and orientation of the information processing apparatus in the constructed real-world coordinate system to the one or more other information processing apparatuses.

2. The information processing apparatus of claim 1, wherein the spatial recognition unit constructs the real-world coordinate system based on parameters for setting coordinates in the real-world coordinate system that are predetermined in association with features of the marker image.

3. The information processing apparatus of claim 1, further comprising:

a sensor that detects an operation input by a user, and outputs a detection signal based on the detection, wherein

the virtual object display unit controls the display unit to display the one or more virtual objects based on the detection signal output by the sensor.

4. The information processing apparatus of claim 3, wherein

the controller, based on a detection signal output from the sensor, controls the communication interface to transmit at least one of the information corresponding to the position and orientation of the information processing apparatus in the constructed real-world coordinate system, data stored in the storage unit, data existing on the network, and information corresponding to a position and orientation of the acquired data in the real-world coordinate system to the one or more information processing apparatuses.

5. The information processing apparatus of claim 1, wherein

the communication interface acquires data transmitted from the one or more other information processing apparatuses, and

the virtual object display unit places the acquired data in the real-world coordinate system based on information corresponding to a position and orientation of the acquired data in the real-world coordinate system received together with the data, and controls the display unit to display the acquired data as one or more virtual objects at positions on the display unit corresponding to the acquired information corresponding to the position and orientation of the acquired data.

6. The information processing apparatus of claim 1, wherein

the controller controls the communication interface to transmit at least one of information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system, data stored in the storage unit, data existing on the network, and information on a position and orientation of the data in the real-world coordinate system to the one or more other information processing apparatuses at predetermined time intervals.

7. The information processing apparatus of claim 3, further comprising:

a graphic object determining unit that determines a graphic object to be displayed as a virtual object by the virtual object display unit according to a detection signal output from the sensor.

8. The information processing apparatus of claim 3, wherein

the virtual object display unit, based on a detection signal output from the sensor, controls the display unit to display the acquired data as the one or more virtual objects in an orientation facing forward with respect to the display unit.

9. The information processing apparatus of claim 3, wherein

the sensor is a touch sensor stacked on or formed integrally with the display unit and detects positions touched by an operation element and outputs the detected positions as position information, and

the virtual object display unit controls a position of a virtual object on the display unit based on the position information output from the touch sensor.

10. The information processing apparatus of claim 1, wherein

the communication interface receives information corresponding to a position and orientation of one of the one or more another information processing apparatuses in the real-world coordinate system, and

the virtual object display unit controls the display unit to display a virtual object representing the one of the one or more another information processing apparatuses based on the received information corresponding to the position and orientation of one of the one or more another information processing apparatuses in the real-world coordinate system.

11. The information processing apparatus of claim 1, wherein

the spatial recognition unit computes a distance to one of the one or more other information processing apparatuses in the real-world coordinate system based on information corresponding to a position and orientation of the one of the one or more other information processing apparatus in the real-world coordinate system received via the communication interface, and

the virtual object display unit controls the display to display a virtual object indicating the distance.

12. The information processing apparatus of claim 1, wherein

the spatial recognition unit computes a distance to one of the one or more other information processing apparatuses in the real-world coordinate system based on information corresponding to a position and orientation of the one of the one or more other information processing apparatuses in the real-world coordinate system received via the communication interface, and

the controller controls the communication interface to transmit data stored in the storage unit or data existing on the network to the one of the one or more other information processing apparatuses when the distance in the real-world coordinate system is less than a predetermined distance.

13. The information processing apparatus of claim 1, wherein

the spatial recognition unit computes a distance between data displayed as a virtual object and the information processing apparatus in the real-world coordinate system based on information corresponding to a position and orientation in the real-world coordinate system of the data displayed as a virtual object received from one of one or more another information processing apparatuses received via the communication interface, and

the controller controls the communication interface to transmit data stored in the storage unit or data existing on the network to the one of the one or more other information processing apparatuses when the distance in the real-world coordinate system between the virtual object being drawn by the one of the one or more information processing apparatuses and the information processing apparatus is less than a predetermined distance.

14. The information processing apparatus of claim 9, further comprising:

a graphic object determining unit that determines a graphic object to be displayed as a virtual object by the virtual object display unit according to a number of inputs detected by the touch sensor, and

the virtual object display unit controls the display unit to display the graphic object as a virtual object at a position where the touch sensor was touched.

15. The information processing apparatus of claim 9, further comprising:

a graphic object determining unit that determines a graphic object to be displayed as a virtual object by the virtual object display unit according to a trail of touch positions detected by the touch sensor, and

the virtual object display unit controls the display to display the graphic object as a virtual object at a position corresponding to where the touch sensor was touched.

16. The information processing apparatus of claim 3, further comprising:

a graphic object determining unit that determines a graphic object to be displayed as a virtual object by the virtual object display unit according to a detection signal output from the sensor, wherein

the sensor is an acceleration sensor or an angular velocity sensor,

the graphic object determining unit changes data to select as a graphic object from among the data stored in the storage unit or the data existing on the network according to a number of times a waving action is performed when it is determined, based on the detection signal output from the acceleration sensor or the angular velocity sensor, that the client device has been waved by the user, and

the virtual object display unit controls the display to display the data selected by the graphic object determining unit as a virtual object at an arbitrary on-screen position on the display unit.

17. The information processing apparatus of claim 1, further comprising:

an operation input unit that accepts an input operation by the user, and outputs an operation signal according to the input operation; and

a graphic object determining unit that determines data displayed as a graphic object based on the operation signal output by the operation input unit, wherein

the virtual object display unit controls the display to display the data selected by the graphic object determining unit as a virtual object at an arbitrary on-screen position on the display unit.

18. The information processing apparatus of claim 1, further comprising:

a microphone that picks up surrounding audio and generates an audio signal; and

a graphic object determining unit that determines data displayed as a graphic object based on features in the audio signal generated by the microphone, wherein

the virtual object display unit controls the display to display the data selected by the graphic object determining unit as a virtual object at an arbitrary on-screen position on the display unit.

19. An information processing method performed by an information processing apparatus, the method comprising:

capturing a subject and outputting image data corresponding to the captured subject;

displaying the captured image;

detecting whether an image preset as a marker image exists in the image data;

constructing, when the marker image is detected in the image data, a real-world coordinate system computed from a position and orientation of the marker image;

computing information corresponding to the information processing apparatus'"'"'s position and orientation in the constructed real-world coordinate system;

acquiring at least one of data stored in a storage unit of the information processing apparatus, data existing on a network, and data transmitted from another information processing apparatus;

placing the acquired data in the constructed real-world coordinate system;

displaying the acquired data as one or more virtual objects at on-screen positions of a display corresponding to positions where the acquired data was placed; and

transmitting information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system to the another information processing apparatus.

20. A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing apparatus, cause the information processing apparatus to perform a process comprising:

capturing a subject and outputting image data corresponding to the captured subject;

displaying the captured image;

detecting whether an image preset as a marker image exists in the image data;

constructing, when the marker image is detected in the image data, a real-world coordinate system computed from a position and orientation of the marker image;

computing information corresponding to the information processing apparatus'"'"'s position and orientation in the constructed real-world coordinate system;

acquiring at least one of data stored in a storage unit of the information processing apparatus, data existing on a network, and data transmitted from another information processing apparatus;

placing the acquired data in the constructed real-world coordinate system;

displaying the acquired data as one or more virtual objects at on-screen positions of a display corresponding to positions where the acquired data was placed; and

transmitting information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system to the another information processing apparatus.