Client device

• US 9,293,118 B2
• Filed: 01/31/2013
• Issued: 03/22/2016
• Est. Priority Date: 03/30/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Mobile Communications Incorporated (Sony Group Corp.)
• Inventors: Matsui, Daisuke
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): ELBINGER, STEVEN Z
• Application Number:
  US13/755,896
• Publication Number:
  US 20130257907A1
• Time in Patent Office: 1,146 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality

Abstract

An information processing apparatus comprising an image recognition unit that detects whether an image preset as a marker image exists in a captured image; a spatial recognition unit that, when the marker image is detected by the image recognition unit, constructs a real-world coordinate system computed from a position and orientation of the marker image, and computes information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system; and a virtual object display unit that acquires at least one of the data stored in the storage unit, data existing on a network, and data transmitted from the one or more other information processing apparatuses and received by the communication interface, places the acquired data in the constructed real-world coordinate system, and controls the display unit to display the acquired data as one or more virtual objects.

Claims

1. An information processing apparatus comprising:

a communication interface that communicates with one or more other information processing apparatuses;

an imaging circuit that captures an image corresponding to a subject and outputs the captured image;

a display circuit that displays the captured image;

a storage circuit storing data;

a sensor that detects an operation input by a user, and outputs a detection signal based on the detection, wherein the sensor comprises a touch sensor stacked on or formed integrally with the display circuit and detects positions touched by an operation element and outputs the detected positions as position information; and

circuitry configured to

detect whether an image preset as a marker image exists in the captured image;

construct, when the marker image is detected, a real-world coordinate system computed from a position and orientation of the marker image, and compute information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system;

acquire at least one of the data stored in the storage circuit, data existing on a network, and data transmitted from the one or more other information processing apparatuses and received by the communication interface, place the acquired data in the constructed real-world coordinate system, and control the display circuit to display the acquired data as one or more virtual objects;

control the communication interface to transmit information corresponding to the position and orientation of the information processing apparatus in the constructed real-world coordinate system to the one or more other information processing apparatuses;

store a correspondence table relating predetermined graphic objects to be displayed with numbers of inputs to be detected by the touch sensor;

determine one or more of the predetermined graphic objects to be displayed according to a number of inputs detected by the touch sensor in accordance with the correspondence table; and

control the display circuit to display the one or more virtual objects based on the position information output from the touch sensor and to display the one or more predetermined graphic objects as one or more of the one or more virtual objects.

2. The information processing apparatus of claim 1, wherein

the circuitry constructs the real-world coordinate system based on parameters for setting coordinates in the real-world coordinate system that are predetermined in association with features of the marker image.

3. The information processing apparatus of claim 1, wherein

the circuitry, based on a detection signal output from the sensor, controls the communication interface to transmit at least one of the information corresponding to the position and orientation of the information processing apparatus in the constructed real-world coordinate system, data stored in the storage circuit, data existing on the network, and information corresponding to a position and orientation of the acquired data in the real-world coordinate system to the one or more information processing apparatuses.

4. The information processing apparatus of claim 1, wherein

the communication interface acquires data transmitted from the one or more other information processing apparatuses, and

the circuitry places the acquired data in the real-world coordinate system based on information corresponding to a position and orientation of the acquired data in the real-world coordinate system received together with the data, and controls the display circuit to display the acquired data as one or more virtual objects at positions on the display circuit corresponding to the acquired information corresponding to the position and orientation of the acquired data.

5. The information processing apparatus of claim 1, wherein

the circuitry controls the communication interface to transmit at least one of information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system, data stored in the storage circuit, data existing on the network, and information on a position and orientation of the data in the real-world coordinate system to the one or more other information processing apparatuses at predetermined time intervals.

6. The information processing apparatus of claim 1, wherein

the circuitry, based on a detection signal output from the sensor, controls the display circuit to display the acquired data as the one or more virtual objects in an orientation facing forward with respect to the display circuit.

7. The information processing apparatus of claim 1, wherein

the communication interface receives information corresponding to a position and orientation of one of the one or more other information processing apparatuses in the real-world coordinate system, and

the circuitry controls the display circuit to display a virtual object representing the one of the one or more other information processing apparatuses based on the information corresponding to the position and orientation of one of the one or more other information processing apparatuses in the real-world coordinate system.

8. The information processing apparatus of claim 1, wherein

the circuitry computes a distance to one of the one or more other information processing apparatuses in the real-world coordinate system based on information corresponding to a position and orientation of the one of the one or more other information processing apparatus in the real-world coordinate system received via the communication interface, and

the circuitry controls the display circuit to display a virtual object indicating the distance.

9. The information processing apparatus of claim 1, wherein

the circuitry computes a distance to one of the one or more other information processing apparatuses in the real-world coordinate system based on information corresponding to a position and orientation of the one of the one or more other information processing apparatuses in the real-world coordinate system received via the communication interface, and

the circuitry controls the communication interface to transmit data stored in the storage circuit or data existing on the network to the one of the one or more other information processing apparatuses when the distance in the real-world coordinate system is less than a predetermined distance.

10. The information processing apparatus of claim 1, wherein

the circuitry computes a distance between data displayed as a virtual object and the information processing apparatus in the real-world coordinate system based on information corresponding to a position and orientation in the real-world coordinate system of the data displayed as a virtual object received from one of one or more other information processing apparatuses received via the communication interface, and

the circuitry controls the communication interface to transmit data stored in the storage circuit or data existing on the network to the one of the one or more other information processing apparatuses when the distance in the real-world coordinate system between the virtual object being drawn by the one of the one or more information processing apparatuses and the information processing apparatus is less than a predetermined distance.

11. The information processing apparatus of claim 1, wherein the circuitry controls the display circuit to display the one or more predetermined graphic objects at a position or positions where the touch sensor was touched.

12. The information processing apparatus of claim 1, wherein:

the circuitry determines the one or more predetermined graphic objects to be displayed according to a trail of touch positions detected by the touch sensor, and

the circuitry controls the display circuit to display the one or more predermined graphic objects at a position or positions corresponding to where the touch sensor was touched.

13. The information processing apparatus of claim 1, wherein:

the circuitry determines the one or more predetermined graphic objects to be displayed according to a detection signal output from the sensor, wherein

the sensor further comprises an acceleration sensor or an angular velocity sensor,

the circuitry changes data to select as the one or more predetermined graphic objects from among the data stored in the storage circuit or the data existing on the network according to a number of times a waving action of a client device is performed when it is determined, based on the detection signal output from the acceleration sensor or the angular velocity sensor, that the client device has been waved by the user, and

the circuitry controls the display circuit to display the data selected at an arbitrary on-screen position on the display circuit.

14. The information processing apparatus of claim 1, wherein:

the circuitry controls the display circuit to display the acquired data at an arbitrary on-screen position on the display circuit.

15. The information processing apparatus of claim 1, further comprising a microphone that picks up surrounding audio and generates an audio signal, wherein

the circuitry controls the display circuit to display the acquired data at an arbitrary on-screen position on the display circuit.

16. An information processing method performed by an information processing apparatus, the method comprising:

capturing a subject and outputting image data corresponding to the captured subject;

displaying the captured image;

detecting whether an image preset as a marker image exists in the image data;

constructing, when the marker image is detected in the image data, a real-world coordinate system computed from a position and orientation of the marker image;

computing information corresponding to the information processing apparatus'"'"'s position and orientation in the constructed real-world coordinate system;

acquiring at least one of data stored in a storage circuit of the information processing apparatus, data existing on a network, and data transmitted from another information processing apparatus;

detecting an operation input by a user through a touch sensor and outputting a detection signal comprising position information indicating positions touched by the user;

placing the acquired data in the constructed real-world coordinate system;

displaying the acquired data as one or more virtual objects at on-screen positions of a display corresponding to positions where the acquired data was placed; and

transmitting information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system to the another information processing apparatus;

wherein said displaying comprises;

storing a correspondence table relating predetermined graphic objects to be displayed with numbers of inputs to be detected by the touch sensor;

determining one or more of the predetermined graphic objects to be displayed according to a number of inputs detected by the touch sensor in accordance with the correspondence table; and

displaying the one or more virtual objects based on the position information output from the touch sensor and displaying the one or more predetermined graphic objects as one or more of the one or more virtual objects.

17. A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing apparatus, cause the information processing apparatus to perform a process comprising:

capturing a subject and outputting image data corresponding to the captured subject;

displaying the captured image;

detecting whether an image preset as a marker image exists in the image data;

constructing, when the marker image is detected in the image data, a real-world coordinate system computed from a position and orientation of the marker image;

computing information corresponding to the information processing apparatus'"'"'s position and orientation in the constructed real-world coordinate system;

acquiring at least one of data stored in a storage circuit of the information processing apparatus, data existing on a network, and data transmitted from another information processing apparatus;

detecting an operation input by a user through a touch sensor and outputting a detection signal comprising position information indicating positions touched by the user;

placing the acquired data in the constructed real-world coordinate system;

displaying the acquired data as one or more virtual objects at on-screen positions of a display corresponding to positions where the acquired data was placed; and

transmitting information corresponding to a position and orientation of the information processing apparatus in the constructed real-world coordinate system to the another information processing apparatus;

wherein said displaying comprises;

storing a correspondence table relating predetermined graphic objects to be displayed with numbers of inputs to be detected by the touch sensor;

determining one or more of the predetermined graphic objects to be displayed according to a number of inputs detected by the touch sensor in accordance with the correspondence table; and

displaying the one or more virtual objects based on the position information output from the touch sensor and displaying the one or more predetermined graphic objects as one or more of the one or more virtual objects.