Information processing apparatus and method, information processing system, and providing medium

• US 9,507,415 B2
• Filed: 10/25/2012
• Issued: 11/29/2016
• Est. Priority Date: 10/19/1998
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Rekimoto, Junichi
• Primary Examiner(s): Theriault, Steven B
• Application Number:
  US13/660,365
• Publication Number:
  US 20130055130A1
• Time in Patent Office: 1,496 Days
• Field of Search: 715/848
• US Class Current: 1/1
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

The invention enables users to virtually attach information to situations in the real world, and also enables users to quickly and easily find out desired information. An IR sensor receives an IR signal transmitted from an IR beacon, and supplies the received signal to a sub-notebook PC. A CCD video camera takes in a visual ID from an object, and supplies the inputted visual ID to the sub-notebook PC. A user inputs, through a microphone, a voice to be attached to situations in the real world. The sub-notebook PC transmits position data, object data and voice data, which have been supplied to it, to a server through a communication unit. The transmitted data is received by the server via a wireless LAN. The server stores the received voice data in a database in correspondence to the position data and the object data.

Claims

1. A wearable user device comprising:

at least one processing unit configured to;

receive image data;

transmit position information indicating a physical position of the wearable user device to a first information processing apparatus;

receive attachment information corresponding to the transmitted position information from a second information processing apparatus via a network; and

control a display in the wearable user device to display the attachment information combined with the image data,

wherein at least part of the attachment information is displayed as a user executable link on the display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

2. A wearable user device according to claim 1, wherein the at least one processing unit is further configured to:

control a communication unit to connect to the network.

3. A wearable user device according to claim 1, wherein the at least one processing unit is further configured to:

control a communication unit to communicate bidirectionally within an augmented reality network system.

4. A wearable user device according to claim 1, wherein the at least one processing unit is further configured to:

control the display to display the user executable link displayed as the icon with an image of the physical position.

5. A wearable user device according to claim 1, wherein the at least one processing unit is further configured to:

input new attachment information to be attached to the position information.

6. A wearable user device according to claim 5, wherein the new attachment information to be attached to the position information is input based on a user operation.

7. A wearable user device according to claim 1, further comprising:

a GPS device,

wherein the at least one processing unit is further configured to;

determine the position information using the GPS device.

8. A wearable user device according to claim 1, wherein the first information processing apparatus and the second information processing apparatus are a same apparatus.

9. A wearable user device according to claim 1, wherein the at least one processing unit is further configured to:

control the display to display additional attachment information based on an execution of the user executable link.

10. A wearable user device according to claim 1, wherein the received image data includes an image of the physical position.

11. A wearable user device according to claim 10, further comprising:

an image input unit,

wherein the image of the physical position is obtained using the image input unit.

12. A wearable user device according to claim 10, wherein the at least one processing unit is further configured to:

control the display to display an image combining the image of the physical position and the user executable link.

13. A wearable user device according to claim 10, wherein the at least one processing unit is further configured to:

superimpose the image data and the user executable link.

14. A method performed by a wearable user device comprising:

receiving image data;

transmitting position information indicating a physical position of the wearable user device to a first information processing apparatus;

receiving attachment information corresponding to the transmitted position information from a second information processing apparatus via a network; and

controlling a display in the wearable user device to display the attachment information combined with the image data,

wherein the attachment information includes a user executable link on the display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

15. A method according to claim 14, further comprising:

connecting to the network.

16. A method according to claim 14, further comprising:

communicating bidirectionally within an augmented reality network system.

17. A method according to claim 14, further comprising:

displaying the user executable link displayed as the icon with an image of the physical position.

18. A method according to claim 14, further comprising:

input new attachment information to be attached to the position information.

19. A method according to claim 18, wherein the new attachment information to be attached to the position information is input based on a user operation.

20. A method according to claim 14, further comprising:

determining the position information using a GPS device included in the wearable user device.

21. A method according to claim 14, wherein the first information processing apparatus and the second information processing apparatus are a same apparatus.

22. A method according to claim 14, further comprising:

displaying additional attachment information based on an execution of the user executable link.

23. A method according to claim 14, wherein the received image data includes an image of the physical position.

24. A method according to claim 23, wherein the image of the physical position is obtained using an image input unit included in the wearable user device.

25. A method according to claim 23, further comprising:

displaying an image combining the image of the physical position and the user executable link.

26. A method according to claim 14, further comprising:

superimposing the image data and the user executable link.

27. A non-transitory computer readable storage medium storing a computer program for causing a wearable user device to:

receive image data;

transmit position information indicating a physical position of the wearable user device to a first information processing apparatus;

receive attachment information corresponding to the transmitted position information from a second information processing apparatus via a network; and

control a display in the wearable user device to display the attachment information combined with the image data,

wherein the attachment information includes a user executable link on the display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

28. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

control a communication unit to connect to the network.

29. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

control a communication unit to communicate bidirectionally within an augmented reality network system.

30. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

control the display to display the user executable link displayed as the icon with an image of the physical position.

31. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

input new attachment information to be attached to the position information.

32. A non-transitory computer readable storage medium according to claim 31, wherein the new attachment information to be attached to the position information is input based on a user operation.

33. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

determine the position information using a GPS device included in the wearable user device.

34. A non-transitory computer readable storage medium according to claim 27, wherein the first information processing apparatus and the second information processing apparatus are a same apparatus.

35. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

control the display to display additional attachment information based on an execution of the user executable link.

36. A non-transitory computer readable storage medium according to claim 27, wherein the received image data includes an image of the physical position.

37. A non-transitory computer readable storage medium according to claim 36, wherein the image of the physical position is obtained using an image input unit included in the wearable user device.

38. A non-transitory computer readable storage medium according to claim 36, further causing the wearable user device to:

control the display to display an image combining the image of the physical position and the user executable link.

39. A non-transitory computer readable storage medium according to claim 27, further causing the wearable user device to:

superimpose the image data and the user executable link.

40. An information processing device comprising:

a storage device configured to store position information indicating a physical position and corresponding attachment information; and

at least one processing unit configured to;

receive position information of a wearable user device from the wearable user device;

transmit the attachment information corresponding to the received position information to the wearable user device via a network,

wherein at least part of the attachment information is displayed as a user executable link on a display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

41. An information processing device according to claim 40, wherein the at least one processing unit is further configured to:

control a communication unit to connect to the network.

42. An information processing device according to claim 40, wherein the at least one processing unit is further configured to:

control a communication unit to communicate bidirectionally within an augmented reality network system.

43. An information processing device according to claim 40, wherein the user executable link is displayed as the icon with an image of a physical position.

44. An information processing device according to claim 40, wherein the at least one processing unit is further configured to:

receive new attachment information to be attached to the position information.

45. An information processing device according to claim 44, wherein the at least one processing unit is further configured to:

store the new attachment information in the storage device to correspond to the position information.

46. An information processing device according to claim 40, wherein an image combining an image of the physical position and the user executable link is displayed.

47. An information processing device according to claim 40, wherein an image superimposing image data and the user executable link is displayed.

48. An information processing method comprising:

storing, by a storage device, position information indicating a physical position and corresponding attachment information;

receiving, by an information processing device, position information of a wearable user device from the wearable user device; and

transmitting the attachment information corresponding to the received position information to the wearable user device via a network,

wherein at least part of the attachment information is displayed as a user executable link on a display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

49. An information processing method according to claim 48, further comprising:

connecting to the network.

50. An information processing method according to claim 48, further comprising:

communicating bidirectionally within an augmented reality network system.

51. An information processing method according to claim 48, wherein the user executable link is displayed as the icon with an image of a physical position.

52. An information processing method according to claim 48, further comprising:

receiving new attachment information to be attached to the position information.

53. An information processing method according to claim 52, further comprising:

storing the new attachment information to correspond to the position information.

54. An information processing method according to claim 48, wherein an image combining an image of the physical position and the user executable link is displayed.

55. An information processing method according to claim 48, wherein an image superimposing image data and the user executable link is displayed.

56. A non-transitory computer readable storage medium storing a computer program for causing an information processing device to:

store position information indicating a physical position and corresponding attachment information;

receive position information of a wearable user device from the wearable user device; and

transmit the attachment information corresponding to the received position information to the wearable user device via a network,

wherein at least part of the attachment information is displayed as a user executable link on a display in the wearable user device,

wherein the display is configured to display a screen image including a context-aware area and a personal information tray, wherein the context-aware area corresponds to a real world associated with the wearable user device, and wherein an icon corresponding to the attachment information is configured to be generated in the personal information tray, wherein a drag and drop operation of the icon is configured to be performed on the display, wherein the icon is configured to be dragged from a first position in the personal information tray and dropped at a second position in the context-aware area, and wherein accessing the icon provides the attachment information.

57. A non-transitory computer readable storage medium according to claim 56, further causing the information processing apparatus to:

connect to the network.

58. A non-transitory computer readable storage medium according to claim 56, further causing the information processing apparatus to:

communicate bidirectionally within an augmented reality network system.

59. A non-transitory computer readable storage medium according to claim 56, wherein the user executable link is displayed as the icon with an image of a physical position.

60. A non-transitory computer readable storage medium according to claim 56, further causing the information processing apparatus to:

receive new attachment information to be attached to the position information.

61. A non-transitory computer readable storage medium according to claim 60, further causing the information processing apparatus to:

store the new attachment information to correspond to the position information.

62. A non-transitory computer readable storage medium according to claim 56, wherein an image combining an image of the physical position and the user executable link is displayed.

63. A non-transitory computer readable storage medium according to claim 56, wherein an image superimposing image data and the user executable link is displayed.