Information processing apparatus to process observable virtual objects

• US 9,772,683 B2
• Filed: 03/17/2010
• Issued: 09/26/2017
• Est. Priority Date: 04/28/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Matsuda, Kouichi
• Primary Examiner(s): Chen, Yu
• Application Number:
  US13/266,301
• Publication Number:
  US 20120092333A1
• Time in Patent Office: 2,750 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A configuration is provided in which data that is not displayed on a display is always observable as an virtual object. For example, in data processing using a PC, data that is not displayed on the display of the PC, such as data having been cut or copied, can be set as a virtual object and pasted on a finger of a hand of a user or the like to be always observable. According to this configuration, data that is not displayed on the display area of the PC can be pasted on a space other than the display area of the PC and displayed to be observable, which allows improving of the efficiency of data processing.

Claims

1. An information processing apparatus, comprising:

a memory configured to contain store a virtual object information management table; and

at least one processor configured to;

obtain display data that indicates a virtual object;

detect a finger, of a hand of a user in a camera-shot image, touched by the user;

analyze a position of the detected finger based on concurrent localization and mapping;

determine whether the position of the detected finger is registered as a display position in the virtual object information management table;

register the position of the detected finger as a position of the obtained display data in the virtual object information management table, wherein the registration is based on the determination that the position of the detected finger is unregistered as the display position;

notify the user that the position of the detected finger is registered as the display position, wherein the notification is based on the determination that the position of the detected finger is registered as the display position; and

inquire to the user whether to delete the display position from the virtual object information management table, wherein the inquiry is based on the determination that the detected finger is registered as the display position.

2. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to control a display screen to display the virtual object to be superimposed at the position of the obtained display data.

3. The information processing apparatus according to claim 2,

wherein the information processing apparatus is a head mounted apparatus, and

wherein the head mounted apparatus comprises the display screen.

4. The information processing apparatus according to claim 2, wherein the at least one processor is further configured to output data, that corresponds to the displayed virtual object, to a data processor of an external device based on a processing request associated with the displayed virtual object.

5. The information processing apparatus according to claim 4, wherein the at least one processor is further configured to obtain, as the display data, data that has been cut or copied from the external device.

6. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to obtain specification information that is input by the user to specify the display position of the virtual object to be deleted from the virtual object information management table.

7. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to register a plurality of positions of fingers, which are touched by the user, as display positions of a plurality of virtual objects in the virtual object information management table.

8. An information processing method, comprising:

obtaining display data indicating a virtual object;

detecting a finger, of a hand of a user in a camera-shot image, touched by the user;

analyzing a position of the detected finger based on concurrent localization and mapping;

determining whether the position of the detected finger is registered as a display position in a virtual object information management table;

registering the position of the detected finger as the display position of the obtained display data in the virtual object information management table, wherein the registration is based on the determination that the position of the detected finger is unregistered as the display position;

notifying the user that the position of the detected finger is registered as the display position, wherein the notification is based on the determination that the detected finger is registered as the display position; and

inquiring to the user whether to delete the display position from the virtual object information management table, wherein the inquiry is based on the determination that the detected finger is registered as the display position.

9. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, cause a computer to perform operations, the operations comprising:

obtaining display data indicating a virtual object;

detecting a finger, of a hand of a user in a camera-shot image, touched by the user;

analyzing a position of the detected finger based on concurrent localization and mapping;

determining whether the position of the detected finger is registered as a display position in a virtual object information management table;

registering the position of the detected finger as a position of the obtained display data in the virtual object information management table, wherein the registration is based on the determination that the position of the detected finger is unregistered as the display position;

notifying the user that the position of the detected finger is registered as the display position, wherein the notification is based on a determination that the touched finger is registered as the display position; and

inquiring to the user whether to delete the display position from the virtual object information management table, wherein the inquiry is based on the determination that the detected finger is registered as the display position.