System and method for transparently processing multimedia data

• US 8,606,950 B2
• Filed: 05/31/2006
• Issued: 12/10/2013
• Est. Priority Date: 06/08/2005
• Status: Expired due to Fees

• Current Assignee: Logitech Europe S A (Logitech International SA)
• Original Assignee: Logitech Europe S A (Logitech International SA)
• Inventors: Glatron, Arnaud, Miauton, Patrick, Bateman, John
• Primary Examiner(s): Moore, Ian N
• Assistant Examiner(s): ZUNIGA ABAD, JACKIE
• Application Number:
  US11/445,497
• Publication Number:
  US 20070214461A1
• Time in Patent Office: 2,750 Days
• Field of Search: 709204-207
• US Class Current: 709/231
• CPC Class Codes: G06F 9/4486 Formation of subprogram jump address
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

A multimedia data processing system and method which transparently processes video and/or audio streams in real-time. The operation of a system in accordance with an embodiment of the present invention does not require any intervention from, or involvement of, either the producer of the video and/or audio stream, or the client application. With such a transparent solution, video and/or audio streams can be processed seamlessly, and completely independently of the specific client application that the user chooses to use. In one embodiment, the invention uses an external service to monitor new processes and add code to these processes. The invention inserts into the system by hot-patching the software in memory and by only looking at select service calls.

Claims

1. A client system for transparently processing multimedia data, comprising:

a data source device for providing multimedia data;

a data sink device for receiving the multimedia data;

a process creation monitor for detecting a plurality of processes created by the client system;

a user mode injection service for injecting with at least one hook at least one process detected wherein hooks are inserted for only a subset of service calls of said process; and

a user mode processing layer on the data sink device to which the at least one hook redirects the multimedia data,

wherein the process creation monitor is configured to notify the user mode injection service for each detected process, and the multimedia data is transparently processed before it reaches the data sink,

wherein the multimedia data that is processed includes audio data and video data, and

wherein said user mode injection service hot patches software in memory of the client system.

2. The system of claim 1 wherein said video data is intercepted at the edge of a kernel, just before the data is sent or received to or from a capture driver.

3. The system of claim 2 wherein said video data is intercepted by:

monitoring device and pin creation requests coming from client applications above;

determining which created pins are of interest;

monitoring traffic to pins of interest by intercepting device I/O Controls sent to these pins; and

stopping monitoring the pins of interest when they are closed.

4. The system of claim 3 wherein said monitoring traffic to pins further comprises:

monitoring creation formats, ‘

set format’

requests and read/write requests.

5. The system of claim 1, wherein the data source is a webcam.

6. The system of claim 1, wherein the data source includes a microphone.

7. The system of claim 1, wherein the multimedia data includes still image data.

8. The system of claim 1, wherein the data sink is a client application.

9. The system of claim 8, wherein the client application is an instant messaging application.

10. The system of claim 1 wherein the multimedia is transparently processed by performing one or more special effects to the multimedia data.

11. The system of claim 10 wherein the special effects include at least one of channel mixing, silence buffer, noise suppression, noise cancellation and notch filtering, distortion, morphing, or spoofing.

12. The system of claim 10 wherein the special effects to be performed are specified by a user.

13. The system of claim 1, wherein the multimedia is transparently processed by performing one or more special effects to the multimedia data, and

wherein the special effects include at least one of color correction, image distortions and alterations, color keying, video avatars, face accessories, stream preview, spoofing, or a rain-drop effect.

14. The system of claim 1 wherein the data sink device is a host device.

15. The system of claim 1 wherein the data source and the data sink are part of the same device.

25. The method of claim 1 wherein the system is a host device.

16. A method for processing multimedia data, wherein the multimedia data is provided by a data source, and the multimedia data is received by a data sink, where the processing is transparent to both the data source and the data sink, the method comprising:

detecting a plurality of processes created in a client system;

notifying a user mode injection service of each of the plurality of detected processes;

injecting, in user mode, at least one hook into a selected process;

inserting said hooks for only a subset of service calls of said selected process;

routing, via the at least one hook, the multimedia data under control of the process to a processing layer;

processing the routed multimedia data in the processing layer in user mode; and

providing the processed multimedia data to the data sink,

wherein the multimedia data that is processed includes audio data and video data, and

wherein said injecting includes providing hot patches to software in memory of the client system.

17. The method of claim 16 wherein said video data is intercepted at the edge of a kernel, just before the data is sent or received to or from a capture driver.

18. The method of claim 17 wherein said video data is intercepted by:

monitoring device and pin creation requests coming from client applications above;

determining which created pins are of interest;

monitoring traffic to pins of interest by intercepting device I/O Controls sent to these pins; and

stopping monitoring the pins of interest when they are closed.

19. The method of claim 18 wherein said monitoring traffic to pins further comprises:

monitoring creation formats, ‘

set format’

requests and read/write requests.

20. The method of claim 16, wherein the multimedia data includes still image data.

21. The method of claim 16 wherein processing the routed multimedia data includes performing one or more special effects to the multimedia data.

22. The method of claim 21 wherein the special effects include at least one of channel mixing, silence buffer, noise suppression, noise cancellation and notch filtering, distortion, morphing, or spoofing.

23. The method of claim 21 wherein the special effects to be performed are specified by a user.

24. The method of claim 16, wherein the multimedia is transparently processed by performing one or more special effects to the multimedia data, and

wherein the special effects include at least one of color correction, image distortions and alterations, color keying, video avatars, face accessories, stream preview, spoofing, or a rain-drop effect.

26. The method of claim 16 wherein the data source is on a different device than the data sink.

27. The method of claim 16 further comprising:

determining whether the process should be hooked; and

injecting, in user mode, at least one hook into the process based on the determination that the process should be hooked.

28. The method of claim 16 wherein said plurality of detected processes includes at least one process that does not include multimedia data.

29. A method for processing multimedia data comprising:

detecting a plurality of processes created in a host system;

notifying a user mode injection service of each of the plurality of detected processes;

injecting, in user mode, at least one hook into a selected process, wherein hooks are inserted for only a subset of service calls of said process;

routing, via the at least one hook, multimedia data under control of the process to a processing layer;

processing, in user mode, the routed multimedia data in the processing layer to perform one or more special effects; and

providing the processed multimedia data to a data sink,

wherein the multimedia data that is processed includes audio data and video data, and

wherein said injecting includes hot patching software in memory of the host system.

30. The method of claim 29 wherein special effects include at least one of color correction, image distortions and alterations, color keying, video avatars, face accessories, stream preview, spoofing, or a rain-drop effect.

31. The method of claim 29 wherein special effects include at least one of channel mixing, silence buffer, noise suppression, noise cancellation and notch filtering, distortion, morphing, or spoofing.

32. The method of claim 29 wherein the one or more special effects are specified by a user.

33. The method of claim 29 wherein said plurality of processes includes at least one process that does not include multimedia data.

34. A client system for transparently processing multimedia data, comprising:

a data source device for providing multimedia data;

a data sink device for receiving the multimedia data;

a process creation monitor for detecting a plurality of processes created by the client system;

a user mode injection service for injecting with at least one hook at least one process detected wherein hooks are inserted for only a subset of service calls of said process; and

a user mode processing layer on the data sink device to process the multimedia data which the at least one hook redirects processed multimedia data,

wherein the process creation monitor is configured to notify the user mode injection service for each detected process, and the multimedia data is transparently processed before it reaches the data sink,

wherein the multimedia data that is processed includes video data and audio data, and

wherein said user mode injection service hot patches software in memory of the client system.