Smart digital modules and smart digital wall surfaces combining the same, and context aware interactive multimedia system using the same and operation method thereof

• US 20050254505A1
• Filed: 02/09/2005
• Published: 11/17/2005
• Est. Priority Date: 05/13/2004
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology, Mit Technology Licensing Office
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Bove, V. Michael Jr., Mullins, Brian J., Chang, Seongju, Mallett, Jacqueline
• Granted Patent: US 7,636,365 B2
• Time in Patent Office: Days
• US Class Current: 370/401
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

Disclosed are smart digital modules, a smart digital wall combining the smart digital modules, a context-aware interactive multimedia system, and an operating method thereof. The smart digital wall includes local coordinators for controlling the smart digital modules, and a coordination process for controlling the smart digital modules. The smart digital modules sense ambient states and changes of states and independently display corresponding actuations. The coordination process is connected to the smart digital modules via radio communication, and combines smart digital modules to control collective operations of the smart digital modules.

Claims

1. A smart digital module intended to work with one or more similar modules for sensing and output of information, comprising:

a sensor unit for sensing ambient states and changes of states;

an actuator unit for outputting various categories of information in correspondence to the ambient states and changes of states;

a display for displaying visual information; and

a computer for processing signals output by the sensor and displaying corresponding actuation information to the actuator and the display,

wherein the computers in a group of smart digital modules coordinate their observations and reactions in such a way that they change states of information output by the actuators and visualized by the displays according to the detected user'"'"'s location and states sensed by the sensors.

2. The smart digital module of claim 1, wherein the sensor unit includes an environment sensor for detecting the ambient states and a behavior sensor for detecting the user'"'"'s changes of activities.

3. The smart digital module of claim 2, wherein the environment sensor includes a temperature sensor for measuring the ambient temperature, a humidity sensor for measuring the ambient humidity, and a luminescence sensor for measuring the ambient luminescence, and

the behavior sensor includes a camera for capturing images, a microphone for receiving speech, a touch sensor for sensing touch inputs, an ultrasonic sensor for sensing the user'"'"'s location through the ultrasonic, and an infrared sensor for sensing infrared signals provided by a remote controller.

4. The smart digital module of claim 1, wherein the computer comprises:

a central processing unit (CPU) for performing data processing and system control;

a memory for temporarily storing data for operation of the CPU and data for various processes;

a storage unit for storing data used by the computer and data input by the sensor unit; and

a display controller for controlling visual information displayed on the display according to control by the CPU.

5. The smart digital module of claim 1, wherein the computer displays a large visual information screen on the display when the user'"'"'s location sensed by the sensor unit is found to be far with reference to the smart digital module.

6. The smart digital module of claim 1, wherein the sensor unit and the actuator unit are provided on a stripe type printed circuit board (PCB) near one of the right and left of the display.

7. The smart digital module of claim 6, wherein the computer is provided on the rear of the display, and

the sensor unit, the actuator unit, the display, and the computer are built into a case.

8. The smart digital module of claim 1, where the user'"'"'s distance and movement are sensed by processing images from the camera.

9. The smart digital module of claim 1, where the user'"'"'s distance and movement are sensed by a combination of processing images from the camera and processing data from other sensors.

10. A smart digital wall for sensing and output of information, comprising:

at least two smart digital modules for sensing ambient states and changes of states and respectively displaying corresponding actuations; and

a coordination process either in a centralized fashion or in a peer-to-peer fashion for combining the smart digital modules and controlling a collective operation of the combined smart digital modules via wired or wireless communication.

11. The smart digital wall of claim 10, where the control process is handled through a coordination process which communicates with all the modules.

12. The smart digital wall of claim 10, where the control process is a peer-to-peer process among the digital modules.

13. The smart digital wall of claim 10, wherein the coordination process realizes adaptive actuations corresponding to sensed events through the smart digital modules based on the user'"'"'s request sensed by at least one smart digital module and recognition of contextual information.

14. The smart digital wall of claim 10, wherein the respective smart digital modules comprise:

a sensor unit for sensing ambient states and changes of states;

an actuator unit for outputting various categories of information in correspondence to the ambient states and changes of states;

a display for displaying visual information; and

a local process for processing signals output by the sensor unit, outputting corresponding actuation information to the actuator unit and the display, and transmitting signals output by the sensor unit to other modules or to a coordination process; and

a wired or wireless LAN unit for transmitting information to other modules or to a coordination process, the LAN unit being connected to the local process,

wherein the coordination process changes the location and format of visual information displayed to the displays installed in the smart digital modules according to the user'"'"'s location sensed by the sensor unit.

15. The smart digital wall of claim 10, wherein the smart digital modules are stacked to form the smart digital wall.

16. The smart digital wall of claim 10, wherein the smart digital modules are inserted into a grid type metallic frame and are interconnected therebetween to form the smart digital wall.

17. The smart digital wall of claim 11, wherein the coordination process is resident on one of the smart digital modules or on another computer or is distributed on all smart digital modules forming the smart digital wall.

18. The smart digital wall of claim 14, wherein when location sensors of the sensor units in the smart digital modules are provided at regular intervals with reference to the smart digital wall, the coordination process uses the location sensors to periodically scan predetermined distances in front of the smart digital wall, measure the user'"'"'s location, and control the location and size of the visual information screen through the actuator units and the displays.

19. The smart digital wall of claim 14, wherein the smart digital modules respectively include a microphone for receiving speech and a speech recognition engine for processing the speech, and

the coordination process processes the user'"'"'s speech acquired through the microphone and the speech recognition engine into words to recognize the user'"'"'s speech instruction.

20. The smart digital wall module of claim 19, wherein the coordination process combines the sounds acquired from microphones on at least two modules before passing the result to the speech recognition process.

21. The smart digital wall of claim 14, wherein the smart digital modules respectively include an infrared sensor for receiving infrared signals, and the coordination process receives infrared signals from the remote controller through the smart digital modules and processes the user'"'"'s instruction.

22. The smart digital wall of claim 14, wherein one of the smart digital modules functions as a web server, and

the web server functions as a web interface for receiving the user'"'"'s instruction from a web server accessible terminal through a network and controlling the smart digital modules.

23. The smart digital wall of claim 14, wherein one of the smart digital modules functions as a server, and

the server functions as a wireless interface for receiving the user'"'"'s instruction from the server accessible mobile terminal including a PDA through a wireless LAN communication, and monitoring and controlling the smart digital modules.

24. The smart digital wall of claim 14, wherein the user'"'"'s front face images are acquired through cameras installed in the smart digital modules, the acquired face images are matched with a plurality of face images stored in a database by using a face recognition program installed in the corresponding smart digital modules to identify the user, and the user'"'"'s identity is used to provide application services in consideration of personal preferences.

25. The smart digital wall of claim 14, wherein the user controls graphic user interface (GUI) components output to a specific smart digital module through touch screens installed in one or more smart digital modules to apply the GUI components as an interface for controlling one or more smart digital modules.

26. The smart digital wall of claim 14, wherein the smart digital modules respectively include a speaker for outputting speech and a speech synthesis engine for synthesizing input text into speech, and

the smart digital modules convert instructions and information provided by the coordination process into speech through the speech synthesis engine, and provide speech instructions and information to the user.

27. The smart digital wall of claim 14, further comprising a database for storing electronic wallpapers generated by combinations of text, images, and video and audio contents,

wherein the coordination process selects a predetermined electronic wallpaper in the database according to an instruction sensed by the sensor units of the smart digital modules or input by the user, and provides a dynamic electronic wallpaper function through the actuators of the smart digital modules and the displays.

28. The smart digital wall of claim 27, wherein when the electronic wallpaper is reproduced in the moving picture format on the smart digital wall, at least one of a location of the electronic wallpaper, a screen size, and a volume of background music is controlled based on the user'"'"'s location and the user'"'"'s instruction sensed by the sensor unit.

29. The smart digital wall of claim 27, wherein a plurality of digital objects including a calendar, a clock, and a picture displayed at predetermined locations together with the electronic wallpaper reproduced on the smart digital wall can be moved, enlarged, reduced, and eliminated according to the user'"'"'s instruction through the sensor units of the smart digital modules.

30. The smart digital wall of claim 14, wherein a contents provider for text, images, and video and audio contents is installed in the smart digital modules, and

the coordination process reproduces multimedia contents input by the contents provider through the sensor units and part or all of the displays according to the instruction sensed by the sensor units or input by the user, provides the reproduced multimedia contents to the user, and thereby provides an interactive multimedia reproducer function.

31. The smart digital wall of claim 30, wherein when the smart digital wall is connected to a remote contents provider providing either or both of a broadcast and a video/audio-on-demand service such as cable, satellite, or Internet provider.

32. The smart digital wall of claim 30, wherein the coordination process performs a reproduction control operation including temporary pause and rewinding of the multimedia contents which are currently reproduced through the user'"'"'s instruction sensed by the sensor units of the smart digital modules.

33. The smart digital wall of claim 14, wherein at least one of the smart digital modules is accessible to an external contents server through the network, and

the coordination process provides information in a user readable format to thereby provide a personalized information browser function, the information being stored in the smart digital modules or downloaded from external contents servers through the network according to the instruction sensed by the sensor units or input by the user.

34. The smart digital wall of claim 33, wherein the coordination process displays information items to the smart digital modules over part or all the displays in a tree structure of hierarchical information according to the instruction sensed by the sensors or input by the user, and displays information items in the lower rank in a like manner when the user selects one of the smart digital modules.

35. The smart digital wall of claim 14, wherein the coordination process analyzes information sensed by the sensors to determine presence of fire, and

when the fire is sensed, the coordination process generates a fire alarm to the actuators and the displays and reports presence of fire to the outside through a communications network to thus provide a smart fire alarm function.

36. The smart digital wall of claim 35, wherein the coordination process displays a flashing red screen to the displays of the smart digital modules when fire has been sensed, to display the emergency of fire to the neighboring users.

37. The smart digital wall of claim 14, wherein the coordination process analyzes information sensed by the sensors to determine whether a person has intruded, and

when the intrusion is found, an intrusion alarm is generated through the actuator units and the displays, and the intrusion is reported to the outside through a communications network to thus provide a smart intrusion security function.

38. The smart digital wall of claim 14, wherein the coordination process analyzes information sensed by the sensors to determine the user'"'"'s health state, and

when the user is found to require medical assistance, an emergency alarm is generated through the actuator units and the displays, and the emergency is reported to the outside through a communications network to thus provide a smart emergency function.

39. A context-aware video chat system based on a user'"'"'s location, comprising:

at least two smart digital modules for respectively measuring a distance to the user, receiving the user'"'"'s video and audio, and outputting the other party'"'"'s video and audio; and

a coordination process for connecting one or more smart digital modules to the other party'"'"'s video chat system through a wired/wireless communications network,

wherein the coordination process controls a number of the smart digital modules for forming a screen on which the other party'"'"'s images are reproduced according to the user'"'"'s location measured by the smart digital modules.

40. The context-aware video chat system of claim 39, wherein the coordination process controls the volume of the other party'"'"'s reproduced speech according to the user'"'"'s location measured by the smart digital modules.

41. The context-aware video chat system of claim 39, wherein the number of smart digital modules reproducing the other party'"'"'s images is greater in the case in which the distance to the user from the context-aware video chat system is longer than a predetermined distance than the case in which the distance to the user from the context-aware video chat system is shorter than the predetermined distance.

42. The context-aware video chat system of claim 40, wherein the speech volume is greater in the case in which the distance to the user from the context-aware video chat system is longer than a predetermined distance than the case in which the distance to the user from the context-aware video chat system is shorter than the predetermined distance.

48. The context-aware video chat system of claim 39, wherein the coordination process processes camera input for finding the user'"'"'s face and thus determines the user'"'"'s distance and movement.

49. The context-aware video chat system of claim 48, wherein the coordination process uses views of the user'"'"'s face from at least two modules, and applies triangulation to determine the user'"'"'s distance.

50. The context-aware video chat system of claim 39, wherein the smart digital modules respectively include:

a sensor unit for measuring the distance to the user, and receiving the user'"'"'s video and audio;

an actuator unit for outputting the other party'"'"'s audio;

a display for outputting the other party'"'"'s video;

a local process for processing signals output by the sensor unit, transmitting the processed signals to other modules or to a coordination process, and displaying the signals transmitted by other modules or a coordination process to the actuator unit and the display; and

a wired or wireless LAN unit for transmitting information to other modules or to a coordination process, the LAN unit being connected to the local process.

51. The context-aware video chat system of claim 39, wherein graphical and/or multimedia information can be displayed on the modules for collaboration purposes in addition to the images of the users.

52. The context-aware video chat system of claim 39, wherein the coordination process is resident on one of the smart digital modules or on another computer.

53. The context-aware video chat system of claim 39, wherein the sensor unit comprises a camera for capturing images, a microphone for receiving speech, a touch sensor for sensing touch inputs, and an ultrasonic sensor for sensing the user'"'"'s location through the ultrasonic.

43. A context-aware video chat system based on a user'"'"'s location, comprising:

at least two smart digital modules for respectively measuring a distance to the user, receiving the user'"'"'s video and audio, and outputting the other party'"'"'s video and audio; and

a coordination process for connecting one or more smart digital modules to the other party'"'"'s video chat system through a wired/wireless communications network,

wherein the coordination process changes the smart digital module on which the other party'"'"'s images are reproduced according to the user'"'"'s movement measured by the smart digital modules.

44. The context-aware video chat system of claim 43, wherein the coordination process changes the smart digital module on which the other party'"'"'s speech is reproduced according to the user'"'"'s movement measured by the smart digital modules.

45. The context-aware video chat system of claim 43, wherein the coordination process uses distance sensors for measuring the distance to the user to perform periodical scanning and sense the user'"'"'s movement.

46. The context-aware video chat system of claim 43, wherein the coordination process controls the number of digital modules which form a screen for reproducing the other party'"'"'s images according to the user'"'"'s location measured by the smart digital modules, and controls the speech volume of the other party.

47. The context-aware video chat system of claim 46, wherein the number of smart digital modules reproducing the other party'"'"'s images is greater in the case in which the distance to the user from the context-aware video chat system is longer than a predetermined distance than the case in which the distance is shorter than the predetermined distance, and

the speech volume is greater in the case in which the distance to the user from the context-aware video chat system is longer than the predetermined distance than the case in which the distance is shorter than the predetermined distance.

54. A context-aware video chat method based on a user'"'"'s location in a context-aware video chat system including at least two smart digital modules for measuring the distance to the user, respectively receiving the user'"'"'s video and audio, and respectively outputting the other party'"'"'s video and audio, and a coordination process connecting one or more smart digital modules to the other party'"'"'s video chat system via a wired/wireless communications network, the method comprising:

(a) controlling the smart digital module reproducing the other party'"'"'s video to measure the distance to the user;

(b) controlling the coordination process to compare the distance measured in (a) with a predetermined distance; and

(c) controlling the coordination process to control the volume of the other party'"'"'s reproduced speech and the screen size occupied by the other party displayed through the smart digital modules according to the comparison result.

55. The context-aware video chat method of claim 54, wherein (c) comprises reducing the screen size to be smaller compared to the case in which the measured distance is longer than the predetermined distance, and reducing the speech volume when the measured distance is shorter than the predetermined distance according to the comparison result.

56. A context-aware video chat method based on a user'"'"'s location in a context-aware video chat system including at least two smart digital modules for measuring the distance to the user, respectively receiving the user'"'"'s video and audio, and respectively outputting the other party'"'"'s video and audio, and a coordination process connecting one or more smart digital modules to the other party'"'"'s video chat system via a wired/wireless network, the method comprising:

(a) controlling the smart digital module reproducing the other party'"'"'s video to periodically measure the distance to the user in a specific order;

(b) controlling the coordination process to determine the video-chatting user'"'"'s movement based on the distance measured in (a); and

(c) controlling the coordination process to determine a smart digital module on which the other party'"'"'s video is reproduced according to the determination result.

57. The context-aware video chat method of claim 56, wherein (b) further comprises controlling the coordination process to compare the distance measured in (a) with a predetermined distance.

58. The context-aware video chat method of claim 56, wherein (c) comprises changing the smart digital module on which the other party'"'"'s video is reproduced when the user is determined to have moved.

59. The context-aware video chat method of claim 57, wherein (c) comprises reducing the screen size to be smaller compared to the case in which the measured distance is longer than the predetermined distance, and reducing the speech volume, when the measured distance is shorter than the predetermined distance according to the comparison result.

60. The context-aware video chat method of claim 56, wherein (a) comprises determining the specific order for smart digital modules with a predetermined height with reference to one or more smart digital modules reproducing the other party'"'"'s video.

61. The context-aware video chat method of claim 56, wherein (a) comprises processing camera input for finding the user'"'"'s face and thus determining the user'"'"'s distance and movement.

62. The context-aware video chat method of claim 61, wherein the image processing uses views of the user'"'"'s face from at least two modules, and applies triangulation to determine the user'"'"'s distance.