Information processing apparatus and information processing method

• US 10,719,121 B2
• Filed: 01/20/2014
• Issued: 07/21/2020
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kimura, Kentaro, Ikeda, Takuo, Ikeda, Tetsuo, Tsukahara, Tsubasa, Koga, Yasuyuki, Izumihara, Atsushi
• Primary Examiner(s): Chatly, Amit
• Application Number:
  US14/772,046
• Publication Number:
  US 20160018885A1
• Time in Patent Office: 2,374 Days
• Field of Search: 345156-179
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An information processing apparatus includes circuitry configured to provide a user interface to a user, determine whether a predetermined object is present in an input recognition area of a sensor, and determine a region of interest within the user interface. The circuitry is further configured to determine, while the predetermined object is determined to be present in the input recognition area, whether the region of interest within the user interface changes. The circuitry is further configured to perform an operation based on whether the region of interest is determined, while the predetermined object is determined to be present in the input recognition area, to change.

Claims

1. An information processing apparatus, comprising:

a first camera configured to capture an image of a plurality of objects in real space;

a storage unit configured to store a plurality of first feature amounts corresponding to the plurality of objects; and

circuitry configured to;

determine a plurality of second feature amounts corresponding to the plurality of objects, wherein the plurality of second feature amounts is determined based on the captured image;

recognize the plurality of objects in the real space based on the plurality of first feature amounts and the plurality of second feature amounts;

control, based on the recognition, a display screen to display a user interface, wherein

the user interface comprises a plurality of virtual objects,

a position of each virtual object of the plurality of virtual objects in the display screen corresponds to a position of a respective object of the recognized plurality of objects in the captured image, and

a posture of each virtual object of the plurality of virtual objects is changeable based on a posture of the respective object of the recognized plurality of objects;

determine, as a user gesture, a presence of a user object in an input recognition area of the first camera;

determine a region of interest within the user interface, wherein the region of interest corresponds to a virtual object of the plurality of virtual objects;

control, based on the determined region of interest, the display screen to display a specific object at a position corresponding to a position of an object of the recognized plurality of objects;

determine that the region of interest remains same during the presence of the user object in the input recognition area;

detect, as a termination of the user gesture, an absence of the user object in the input recognition area;

execute one of a tap operation, a click operation, or a select operation on the specific object based on

the determination that the region of interest remains same and

the detected absence of the user object in the input recognition area; and

control the display screen to display specific information on the specific object, wherein

the display of the specific information on the specific object is based on the detected absence of the user object in the input recognition area and the execution of the one of the tap operation, the click operation, or the selection operation, and

the specific information includes at least one of a title associated with the specific object or contents associated with the specific object.

2. The information processing apparatus according to claim 1, further comprising a sensor configured to generate data representative of a user viewing direction, wherein

the circuitry is further configured to determine the region of interest within the user interface based on the generated data.

3. The information processing apparatus according to claim 2, wherein the sensor is one of a second camera, an acceleration sensor, or a gyro sensor.

4. The information processing apparatus according to claim 1, wherein

the circuitry is further configured to:

determine a change in the region of interest within the user interface based on the user gesture; and

execute a drag operation on the specific object based on the change of the region of interest.

5. The information processing apparatus according to claim 4, wherein the circuitry is further configured to control, based on the change in the region of interest, a first device to transmit content information to a second device.

6. The information processing apparatus according to claim 5, wherein the first device is a television.

7. The information processing apparatus according to claim 1, wherein the circuitry is further configured to transmit, based on the user gesture, a request to a first device to send determined data to the information processing apparatus.

8. The information processing apparatus according to claim 7, wherein the circuitry is further configured to receive the determined data from the first device based on the request.

9. The information processing apparatus according to claim 7, wherein the circuitry is further configured to receive a discount coupon from the first device based on the request.

10. The information processing apparatus according to claim 1, wherein

the circuitry is further configured to:

determine a change in the region of interest within the user interface based on the user gesture; and

execute a drag operation on the specific object based on an amount of the change of the region of interest.

11. The information processing apparatus according to claim 1, wherein the first camera is further configured to capture an image of the user object.

12. The information processing apparatus according to claim 1, further comprising the display screen, wherein the display screen is in a head-mounted display unit.

13. The information processing apparatus according to claim 1, wherein the circuitry is further configured to determine, as the user gesture, a gesture of movement of a part of an operation body.

17. The information processing apparatus according to claim 1, wherein the circuitry is further configured to:

determine a change in the region of interest within the user interface based on the user gesture;

select a specific region of the user interface based on the change in the region of interest;

select the plurality of virtual objects based on the selected specific region;

sort the plurality of virtual objects; and

control the display screen to display the sorted plurality of virtual objects.

14. An information processing system, comprising:

a camera configured to capture an image of a plurality of objects in real space;

a storage unit configured to store a plurality of first feature amounts corresponding to the plurality of objects; and

circuitry configured to;

determine a plurality of second feature amounts corresponding to the plurality of objects, wherein the plurality of second feature amounts is determined based on the captured image;

recognize the plurality of objects in the real space based on the plurality of first feature amounts and the plurality of second feature amounts;

control, based on the recognition, a display screen to display a user interface, wherein

the user interface comprises a plurality of virtual objects,

a position of each virtual object of the plurality of virtual objects in the display screen corresponds to a position of a respective object of the recognized plurality of objects in the captured image, and

a posture of each virtual object of the plurality of virtual objects is changeable based on a posture of the respective object of the recognized plurality of objects;

determine, as a user gesture, a presence of a user object in an input recognition area of the camera;

determine a region of interest within the user interface, wherein the region of interest corresponds to a virtual object of the plurality of virtual objects;

control, based on the determined region of interest, the display screen to display a specific object at a position corresponding to a position of an object of the recognized plurality of objects;

determine that the region of interest remains same during the presence of the user object in the input recognition area;

detect, as a termination of the user gesture, an absence of the user object in the input recognition area;

execute one of a tap operation, a click operation, or a select operation on the specific object based on

the determination that the region of interest remains same and

the detected absence of the user object in the input recognition area; and

control the display screen to display specific information on the specific object, wherein

the display of the specific information on the specific object is based on the detected absence of the user object in the input recognition area and the execution of the one of the tap operation, the click operation, or the selection operation, and

the specific information includes at least one of a title associated with the specific object or contents associated with the specific object.

15. A method, comprising:

capturing, by a camera of an information processing apparatus, an image of a plurality of objects in real space;

storing, in a storage unit of the information processing apparatus, a plurality of first feature amounts corresponding to the plurality of objects;

determining, by circuitry of the information processing apparatus, a plurality of second feature amounts corresponding to the plurality of objects, wherein the plurality of second feature amounts is determined based on the captured image;

recognizing, by the circuitry, the plurality of objects in the real space based on the plurality of first feature amounts and the plurality of second feature amounts;

controlling, by the circuitry, based on the recognition, a display screen to display a user interface, wherein

the user interface comprises a plurality of virtual objects,

a position of each virtual object of the plurality of virtual objects in the display screen corresponds to a position of a respective object of the recognized plurality of objects in the captured image, and

a posture of each virtual object of the plurality of virtual objects is changeable based on a posture of the respective object of the recognized plurality of objects;

determining, by the circuitry, a presence of a user object in an input recognition area of the camera, as a user gesture;

determining, by the circuitry, a region of interest within the user interface, wherein the region of interest corresponds to a virtual object of the plurality of virtual objects;

controlling, by the circuitry, based on the determined region of interest, the display screen to display a specific object at a position corresponding to a position of an object of the recognized plurality of objects;

determining, by the circuitry, that the region of interest remains same during the presence of the user object in the input recognition area;

detecting, by the circuitry, an absence of the user object in the input recognition area, as a termination of the user gesture;

executing, by the circuitry, one of a tap operation, a click operation, or a select operation on the specific object based on

the determination that the region of interest remains same and

the detected absence of the user object in the input recognition area; and

controlling, by the circuitry, the display screen to display specific information on the specific object, wherein

the display of the specific information on the specific object is based on the detected absence of the user object in the input recognition area and the execution of the one of the tap operation, the click operation, or the selection operation, and

the specific information includes at least one of a title associated with the specific object or contents associated with the specific object.

16. A non-transitory computer-readable storage medium having stored thereon, computer-executable instructions which when executed by a processor of an information processing apparatus, cause the processor to execute operations, the operations comprising:

controlling a camera of the information processing apparatus to capture an image of a plurality of objects in real space;

controlling a storage unit of the information processing apparatus to store a plurality of first feature amounts corresponding to the plurality of objects;

determining a plurality of second feature amounts corresponding to the plurality of objects, wherein the plurality of second feature amounts is determined based on the captured image;

recognizing the plurality of objects in the real space based on the plurality of first feature amounts and the plurality of second feature amounts;

controlling, based on the recognition, a display screen to display a user interface, wherein

the user interface comprises a plurality of virtual objects,

a position of each virtual object of the plurality of virtual objects in the display screen corresponds to a position of a respective object of the recognized plurality of objects in the captured image, and

a posture of each virtual object of the plurality of virtual objects is changeable based on a posture of the respective object of the recognized plurality of objects;

determining, as a user gesture, a presence of a user object in an input recognition area of the camera;

determining a region of interest within the user interface, wherein the region of interest corresponds to a virtual object of the plurality of virtual objects;

controlling, based on the determined region of interest, the display screen to display a specific object at a position corresponding to a position of an object of the recognized plurality of objects;

determining that the region of interest remains same during the presence of the user object in the input recognition area;

detecting, as a termination of the user gesture, an absence of the user object in the input recognition area;

executing one of a tap operation, a click operation, or a select operation on the specific object based on

the determination that the region of interest remains same and

the detected absence of the user object in the input recognition area; and

controlling the display screen to display specific information on the specific object, wherein

the display of the specific information on the specific object is based on the detected absence of the user object in the input recognition area and the execution of the one of the tap operation, the click operation, or the selection operation, and

the specific information includes at least one of a title associated with the specific object or contents associated with the specific object.