Information processing apparatus and information processing method, display apparatus and display method, and information processing system

• US 10,269,180 B2
• Filed: 03/04/2014
• Issued: 04/23/2019
• Est. Priority Date: 04/16/2013
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Yamamoto, Kazuyuki, Hosoda, Yasuhide, Kii, Manabu
• Primary Examiner(s): Thompson, James A
• Application Number:
  US14/782,638
• Publication Number:
  US 20160093107A1
• Time in Patent Office: 1,876 Days
• Field of Search: None
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

Interaction between a virtual object and the real space is to be presented in a preferred manner.

In accordance with a user'"'"'s viewing or finding of an actual object 151, the virtual object 150 corresponding to the actual object 151 appears. The actual object 151 includes output devices that generate outputs such as vibration, electric pulse, heat or cooling air, wind, sound, light, a transfer, and a jump. The actual object 151 performs reaction to action performed on the actual object 151 by the virtual object 150. Where interaction is conducted between the virtual object 150 and the actual object 151 in synchronization, a stronger impression of reality can be given to the user.

Claims

1. An information processing apparatus, comprising:

an actual object comprising;

an identifier includes identification information associated with identification of the actual object;

an output unit configured to apply a first action to the actual object based on the identification of the actual object;

a control unit configured to control an output from the output unit based on one of a second action executed on the actual object by a virtual object, or a third action to be executed on the virtual object by the actual object; and

a virtual object control unit configured to change the second action executed on the actual object by the virtual object,

wherein the second action is changed based on at least one of environmental information that comprises information of a real space that surrounds the virtual object and a user, or state information that comprises the user'"'"'s current state in the real space.

2. The information processing apparatus according to claim 1, further comprising

a receiving unit

configured to receive a result of detection of the second action executed on the actual object by the virtual object,

wherein the control unit is further configured to control the output from the output unit based on the received result of the detection of the second action.

3. The information processing apparatus according to claim 1, wherein the control unit is further configured to control the output from the output unit based on the second action executed on the actual object by the virtual object controlled by the virtual object control unit.

4. The information processing apparatus according to claim 1,

wherein the output unit includes at least one of a vibrating device, a pulse generating device, a heat generating device, a cooling device, an air blowing device, an acoustic device, a light emitting device, or a transfer device.

5. The information processing apparatus according to claim 1, wherein one of the second action executed on the actual object by the virtual object or the third action to be executed on the virtual object by the actual object is determined based on a user operation with respect to the virtual object.

6. An information processing method, comprising:

in an information processing apparatus that comprises an actual object that includes an output unit and an identifier having identification information associated with identification of the actual object;

acquiring one of a first action executed on the actual object by a virtual object or a second action executed on the virtual object by the actual object, based on the identification of the actual object;

applying, via the output unit, a third action to the actual object based on one of the first action executed on the actual object by the virtual object or the second action to be executed on the virtual object by the actual object; and

changing the first action executed on the actual object by the virtual object,

wherein the second action is changed based on at least one of environmental information that comprises information of a real space surrounding the virtual object and a user, or state information that comprises the user'"'"'s current state in the real space.

7. A display apparatus, comprising:

a detecting unit configured to;

detect a specific actual object;

read identification information from an identifier included in the specific actual object, based on the detection of the specific actual object; and

identify the specific actual object based on the read identification information;

a display unit configured to display a virtual object based on the identification of the specific actual object;

an input operating unit configured to determine a first action of the virtual object on the specific actual object;

a virtual object control unit configured to change the first action of the virtual object on the specific actual object based on at least one of environmental information comprising information of a real space surrounding the virtual object and a user, or state information comprising the user'"'"'s current state in the real space; and

a location/posture detecting unit configured to detect a location and a posture of one of the user'"'"'s head or the user'"'"'s facial part,

wherein the display unit is further configured to change a display position of the displayed virtual object based on the detected location and the detected posture of the one of the user'"'"'s head or the user'"'"'s facial part.

8. The display apparatus according to claim 7, wherein

the detecting unit is further configured to detect the specific actual object based on the user'"'"'s field of view, and

the display unit is further configured to display the virtual object overlapped on the specific actual object.

9. The display apparatus according to claim 7, wherein the virtual object control unit is further configured to control one of appearance or disappearance of the virtual object based on the user'"'"'s motion.

10. The display apparatus according to claim 7,

wherein the virtual object control unit is further configured to control information about the virtual object displayed on the display unit, and

wherein the information is controlled based on one of the user'"'"'s motion,

the user'"'"'s current state, or a time of day.

11. The display apparatus according to claim 7, wherein the virtual object control unit is further configured to control the first action of the virtual object on the specific actual object, based on a second action received by the virtual object from the specific actual object.

12. The display apparatus according to claim 7, wherein

the detecting unit is further configured to detect at least one of the first action of the virtual object on the specific actual object or a second action received by the virtual object from the specific actual object, and

the virtual object control unit is further configured to control the first action of the virtual object on the specific actual object based on a result of the detection of the at least one of the first action of the virtual object on the specific actual object or the second action received by the virtual object from the specific actual object.

13. The display apparatus according to claim 7, wherein the virtual object control unit is further configured to control the first action of the virtual object on the specific actual object such that the first action of the virtual object on the specific actual object is synchronized with a second action of the specific actual object on the virtual object.

14. The display apparatus according to claim 7,

wherein the display unit is on one of the user'"'"'s head or the user'"'"'s facial part.

15. A display method, comprising:

detecting a specific actual object;

reading identification information from an identifier included in the specific actual object;

identifying the specific actual object based on the identification information;

displaying a virtual object based on the identification of the specific actual object;

changing an action executed on the specific actual object by the virtual object based on at least one of environmental information that comprises information of a real space surrounding the virtual object and a user, or state information that comprises the user'"'"'s current state in the real space;

detecting a location and a posture of one of the user'"'"'s head or the user'"'"'s facial part; and

changing a display position of the virtual object based on the detected location and the detected posture of the one of the user'"'"'s head or the user'"'"'s facial part.

16. An information processing system, comprising:

a display apparatus configured to;

detect an actual object;

read identification information from an identifier included in the actual object based on the detection of the actual object;

identify the actual object based on the read identification information;

display a virtual object corresponding to the actual object based on the identification of the actual object;

detect a location and a posture of one of a user'"'"'s head or the user'"'"'s facial part; and

change a display position of the virtual object based on the detected location and the detected posture of the one of the user'"'"'s head or the user'"'"'s facial;

a control apparatus configured to control a first action of the virtual object on the actual object; and

an output apparatus configured to apply a second action to the actual object based on one of the first action executed of the virtual object on the actual object or a third action to be executed on the virtual object by the actual object,

wherein the display apparatus is further configured to change the first action of the virtual object on the actual object based on at least one of environmental information comprising information of a real space that surrounds the virtual object and the user, or state information comprising the user'"'"'s current state in the real space.

17. An information processing system, comprising:

a display apparatus configured to;

detect an actual object;

read identification information from an identifier included in the actual object based on the detection of the actual object;

identify the actual object based on the read identification information;

display a virtual object corresponding to the actual object based on the identification of the actual object;

control a first action of the virtual object on the actual object based on at least one of environmental information comprising information of a real space that surrounds the virtual object and a user, or state information comprising the user'"'"'s current state in the real space;

detect a location and a posture of one of the user'"'"'s head or the user'"'"'s facial part; and

change a display position of the virtual object based on the detected location and the detected posture of the one of the user'"'"'s head or the user'"'"'s facial part; and

an output apparatus configured to apply a second action to the actual object based on one of the first action executed on the actual object by the virtual object, or a third action to be executed on the virtual object by the actual object.