System and method for indexing, accessing and retrieving audio/video with concurrent sketch activity

• US 6,724,918 B1
• Filed: 05/09/2000
• Issued: 04/20/2004
• Est. Priority Date: 05/12/1999
• Status: Expired due to Term

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Inventors: Fruchter, Renate, Leifer, Larry, Yen, Samuel
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/568,090
• Time in Patent Office: 1,442 Days
• Field of Search: 382/113, 382/100, 382/119, 382/179, 382/168, 382/187, 379/487, 379/93.19, 379/93.08, 709/205, 723/90, 704/231, 715/512, 715/500.1, 348/14.09, 348/14.02, 707/104.1, 345/866, 700/146
• US Class Current: 382/113
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

For a number of users is a system provided to create, edit, replay and view documents of free hand drawn sketches. The system captures the creation process together with verbal and/or visual information provided by each user and automatically correlates them for a later synchronized replay. The system provides a number of tools and features, mainly to: combine the sketching activity with existing images, to selectively retrieve media information correlated to individual sketch entities and to quasi simultaneously collaborate at a common document. The system architecture can by adjusted to various parameters in the communication infrastructure. The system may be implemented in any software program, a web based service, a web browser, an operating system for computers and/or communication devices.

Claims

1. A system for identifying sketch entities from captured sketching activity and correlating said sketch entities with media information, said media information being simultaneously captured with said sketching activity, said system comprising:

a) a drawing recognition means for capturing drawing movements of said sketching activity;

b) an input recognition means for capturing initiation events and termination events of said sketching activity;

c) a time stamping means for time stamping said sketching activity and said media information;

d) a processing means for said identifying of said sketch entities;

e) a database for storing a number of permanent logs;

f) a script log for storing all data of each document created on one of a number of user units; and

g) a distribution means for distributing said permanent logs and one or more script logs between said database and said number of user units.

2. The system of claim 1, wherein said system further comprises:

a computer product comprising a computer readable medium carrying computer-executable program code for providing;

I) said drawing recognition means;

II) said input recognition means;

III) said time stamping means;

IV) said processing means;

V) an interactive graphical interface;

VI) a creating mode;

VII) an editing mode;

VIII) a replay mode;

IX) a viewing mode; and

X) said script log.

3. The system of claim 2, wherein at least one of said number of user units is a computer.

4. The system of claim 2, wherein at least one of said number of user units is a communication device.

5. The system of claim 4, wherein said communication device is a wireless communication device.

6. The system of claim 2, wherein said system further comprises a server.

7. The system of claim 6, wherein said database is maintained by said server.

8. The system of claim 2, wherein said media information contains an audio-signal.

9. The system of claim 2, wherein said media information contains a video-signal.

10. The system of claim 2, wherein at least one of said sketch entities is started with one of said initiation events and is ended with one of said termination events.

11. The system of claim 10, wherein at least one of said initiation events defines a replay starting moment of said media information.

12. The system of claim 2, wherein said script log contains a creating history of said document created on one of said number of user units.

13. The system of claim 2, wherein at least one of said number of permanent logs contains said creating history of one or more of said number of user units.

14. The system of claim 2, wherein said script log contains an editing history of said document, said document being edited on one of said number of user units.

15. The system of claim 2, wherein at least one of said number of permanent logs contains said editing history of one or more of said number of user units.

16. The system of claim 2, wherein said distribution system is a centralistic distribution system.

17. The system of claim 16, wherein said centralistic distribution system is based on a web page.

18. The system of claim 17, wherein said program code is provided on at least one of said user units via said web page.

19. The system of claim 2, wherein said program code is part of a web browser.

20. The system of claim 2, wherein said program code is part of an operating system, said operating system operating at least one of said user units.

21. The system of claim 2, wherein said program code is a self extracting file transmitted to at least one of said user units.

22. The system of claim 21, wherein said self extracting file is in an email attachment.

23. The system of claim 2, wherein said program code is stored in the form of a computer readable code on a direct access storage device of at least one of said user units.

24. The system of claim 2, wherein said program code further provides a background image on said interactive graphical interface.

25. The system of claim 24, wherein said background image is a snapshot derived from said video signal.

26. The system of claim 2, wherein said distribution system is an equalized distribution system.

27. The system of claim 26, wherein said database is stored in form of multiple representations on a number of direct access storage devices of a number of said user units.

28. The system of claim 2, wherein said system provides a time independent communication mode.

29. The system of claim 28, wherein said system provides a low level script log distribution management.

30. The system of claim 2, wherein said system provides a quasi real time communication mode.

31. The system of claim 30, wherein said system provides a high level script log distribution management.

32. A system for identifying sketch entities from captured sketching activity and for identifying media blocks from captured media information, said media information being simultaneously captured with said sketching activity, said system comprising:

a) a drawing recognition means for capturing drawing movements of said sketching activity;

b) an input recognition means for capturing initiation actions and termination actions of said sketching activity;

c) an audio level recognition means for recognizing an audio initiation moment and an audio termination moment;

d) a time stamping means for time stamping said sketching activity and said media information;

e) a processing means for said identifying of said sketch to entities and of said media blocks;

f) a database for storing a number of permanent logs;

g) a script log for storing all data of each document created on one of a number of user units; and

h) a distribution means for distributing said permanent logs and one or more script logs between said database and said number of user units.

33. The system of claim 32, wherein said system further comprises:

a computer product comprising a computer readable medium carrying computer-executable program code for providing;

I) said drawing recognition means;

II) said input recognition means;

III) said time stamping means;

IV) said processing means;

V) an interactive graphical interface;

VI) a creating mode;

VII) an editing mode;

VIII) a replay mode;

IX) a viewing mode;

X) said script log.

34. The system of claim 33, wherein at least one of said number of user units is a computer.

35. The system of claim 33, wherein at least one of said number of user units is a communication device.

36. The system of claim 35, wherein said communication device is a wireless communication device.

37. The system of claim 33, wherein said system further comprises a server.

38. The system of claim 37, wherein said database is maintained by said server.

39. The system of claim 33, wherein said media information contains an audio-signal.

40. The system of claim 33, wherein said media information contains a video-signal.

41. The system of claim 33, wherein at least one of said sketch entities is started with one of said initiation events and is ended with one of said termination events.

42. The system of claim 41, wherein at least one of said initiation events is in block correlation to said audio initiation moment and at least one of said termination events is in block correlation to said audio termination moment of said media block, wherein said audio initiation moment is in level correlation to a first noise switching level, and wherein said audio termination moment is in level correlation to a second noise switching level.

43. The system of claim 42, wherein said block correlation and said level correlation is processed by an approximation algorithm.

44. The system of claim 33, wherein said script log contains a creating history of said document created on one of said number of user units.

45. The system of claim 33, wherein at least one of said number of permanent logs contains said creating history of one or more of said number of user units.

46. The system of claim 33, wherein said script log contains an editing history of said document, said document being edited at one of said number of user units.

47. The system of claim 33, wherein at least one of said number of permanent logs contains said editing history of one or more of said number of user units.

48. The system of claim 33, wherein said distribution system is a centralistic distribution system.

49. The system of claim 48, wherein said centralistic distribution system is based on a web page.

50. The system of claim 49, wherein said program code is provided on at least one of said user units via said web page.

51. The system of claim 33, wherein said program code is part of a web browser.

52. The system of claim 33, wherein said program code is part of an operating system, said operating system operating at least one of said user units.

53. The system of claim 33, wherein said program code is a self extracting file transmitted to at least one of said user units.

54. The system of claim 53, wherein said self extracting file is in an email attachment.

55. The system of claim 33, wherein said program code is stored in the form of a computer readable code on a direct access storage device of at least one of said user units.

56. The system of claim 33, wherein said program code further provides a background image on said interactive graphical interface.

57. The system of claim 56, wherein said background image is a snapshot derived from said video signal.

58. The system of claim 33, wherein said distribution system is an equalized distribution system.

59. The system of claim 58, wherein said database is stored in form of multiple representations on a number of direct access storage devices of a number of said user units.

60. The system of claim 33, wherein said system provides a time independent communication mode.

61. The system of claim 60, wherein said system provides a low level script log distribution management.

62. The system of claim 33, wherein said system provides a quasi real time communication mode.

63. The system of claim 62, wherein said system provides a high level script log distribution management.

64. A method for identifying a sketch entity from captured sketching activity and correlating said sketch entity with a media information, said media information being simultaneously captured with said sketching activity, said method comprising the steps of:

a) time stamped capturing an initiation event of said sketching activity;

b) time stamped capturing drawing movements of said sketching activity;

c) time stamped capturing a termination event of said sketching activity;

d) simultaneously time stamped capturing said media information; and

e) identifying said sketch entity from said initiation event and said termination event;

f) storing all data of each document created on one of a number of user units;

g) storing a number of documents in a database; and

h) distributing said number of documents between said database and said number of user units.

65. The method of claim 64, whereby said media information is an audio information.

66. The method of claim 64, whereby said media information is a video information.

67. A method for identifying a sketch entity from captured sketching activity and for identifying a media block from captured media information, said media information being simultaneously captured with said sketching activity, said method comprising the steps of:

a) time stamped capturing an initiation event of said sketching activity;

b) time stamped capturing drawing movements of said sketching activity;

c) time stamped capturing a termination event of said sketching activity;

d) simultaneously time stamped capturing an audio initiation moment;

e) simultaneously time stamped capturing said media information;

f) simultaneously time stamped capturing an audio termination moment;

g) identifying said media block from said audio initiation moment and said audio termination moment;

h) identifying said sketch entity from said initiation event and said termination event; and

i) correlating said sketch entity to said media block;

j) storing all data of each document created on one of a number of user units;

k) storing a number of documents in a database; and

l) distributing said number of documents between said database and said number of user units.

68. The method of claim 67, whereby said media information is an audio information.

69. The method of claim 67, whereby said media information is a video information.