Selective capture with rapid sharing of user or mixed reality actions and states using interactive virtual streaming

• US 10,419,510 B2
• Filed: 06/20/2018
• Issued: 09/17/2019
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Louis Gurtowski
• Original Assignee: Louis Gurtowski
• Inventors: Gurtowski, Louis
• Primary Examiner(s): Keller, Michael A
• Application Number:
  US16/013,244
• Publication Number:
  US 20180302453A1
• Time in Patent Office: 454 Days
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/266 Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/632 using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices broadcast-related systems characterised by transmission among terminal devices H04H60/80; distributed application using peer-to-peer [P2P] networks H04L67/104

Abstract

A method for streaming interactive media between clients includes streaming a first series of packets from a server to a first client. The packets include data, instructions and timestamps for executing the instructions. At least one of the instructions requests capture of client-based events. A second series of packets is received from the first client device. The second series of packets includes one or more captured client-based events generated in response to the instructions requesting capture of one or more client-based events. The second series of packets further includes one or more timestamps associated with each of the client-based events. The first series of packets is merged with the second series of packets based on the time-stamps included with the first and second series of packets to form a merged series of packets. The merged series of packets are streamed to a second client device.

Claims

1. A method for streaming interactive media between clients, comprising:

streaming from a server to at least a first client device a first series of packets, the packets including data, instructions and timestamps for executing the instructions, at least one of the instructions in at least one of the packets requesting capture of one or more client-based events that are to be communicated back to the server;

receiving from the first client device a second series of packets that include one or more captured client-based events generated in response to the instructions requesting capture of one or more client-based events, the second series of packets further including one or more timestamps associated with the each of the client-based events;

integrating a test pattern into the first series of packets to further validate a timing accuracy of the packets in the second series of packets that include the one or more client-based events;

merging the first series of packets with the second series of packets based on the time-stamps included with the first and second series of packets to form a merged series of packets;

streaming the merged series of packets to a second client device; and

wherein the instructions in the first series of packets include time stamped Remote Procedure Calls (RPC) that include time stamps, procedure identification with a request ID.

2. The method of claim 1 further comprising:

receiving from the first client device a third series of packets that include one or more captured client-based events that are asynchronously generated and not generated in response to the instructions requesting capture of one or more client-based events, the client-based packets further including one or more timestamps associated with the each of the client-based events; and

merging the first, second and third series of packets based on the time-stamps included with the first, second and third series of packets and the client-based packets to form the merged series of packets.

3. The method of claim 1 wherein the captured client-based events in the second series of packets are deliverable to one or more third parties that are able to interactively modify capture requests and parameters in the first series of packets sent by the server and rerun in an entirety the streaming, receiving and merging.

4. The method of claim 3 wherein the third series of client-based packets are deliverable to one or more third parties that are able to rerun in an entirety the streaming, receiving, and merging.

5. The method of claim 1 wherein the time stamped RPCs include at least one parameter selected from the group including a client ID, session ID, one or more objects IDs and one or more data buffers not identified by an ID.

6. The method of claim 1 wherein merging the first and second series of packets further includes removing selected entries from the client-based instructions before streaming the merged series of packets to the second client.

7. The method of claim 6 wherein the selected entries include the user and/or session identifier.

8. The method of claim 1 further comprising:

receiving from the second client device a fourth series of packets that include one or more captured client-based events generated in response to the instructions requesting capture of one or more client-based events, the fourth series of packets further including one or more timestamps associated with the each of the client-based events;

merging the fourth series of packets with the first and second series of packets based on the time-stamps included with the first, second and fourth series of packets to form the merged series of packets.

9. The method of claim 1 wherein the client-based event further includes information reflective of one or more states of the first client device.

10. The method of claim 1 wherein streaming the first series of packets to the client further includes streaming the first series of packets in accordance with an adaptive streaming technique.

11. The method of claim 10 further comprising adaptively streaming the first series of packets to the client to dynamically match a data rate of the second series of packets being received from the first client device.

12. The method of claim 1 wherein the data included in the first series of packets includes at least one interactive media object.

13. The method of claim 1 wherein the one more timestamps included with the second series of packets is generated by a clock that also controls playback of the first series of packets on the first client device.

14. The method of claim 1 wherein the merging further includes converting the merged series of packets to a video format that conforms to a standard video protocol.

15. The method of claim 1 wherein the instructions requesting capture of one or more client-based events include one or more parameters specifying recording settings used for capturing the client-based events.

16. The method of claim 1 wherein the captured client-based events are stored in a buffer and the buffered client-based events are periodically formatted for streaming to the server.

17. The method of claim 1 wherein merging the first and second series of packets includes sorting the instructions in an order based at least in part on the timestamps included with the first and second series of packets.

18. The method of claim 1 wherein merging the first and second series of packets includes translating the client-based events into procedure calls executable by the second client device.

19. The method of claim 1 wherein the client-based events include user-input received by the first client device.

20. The method of claim 1 wherein the client-based events include sensor data provided by one or more sensors associated with the first client device.

21. The method of claim 1 further comprising dynamically revising the instructions based at least in part on the second series of packets received from the first client device.

22. The method of claim 1 further comprising calculating a playback timing error for the first streamed series of packets using timing information sent in the second series of packets and disregarding any of the second series of packets if the playback timing error is not within an allowed range.

23. The method of claim 1 wherein the merged series of packets is sent by the server to one or more observer-clients.

24. A method for streaming interactive multimedia between clients, comprising

streaming from a server to a first producer-client and a second producer-client a first series of packets, the packets including data, instructions and timestamps for executing the instructions, at least one of the instructions in at least one of the packets requesting capture of one or more client-based events that are to be communicated back to the server;

receiving separately from the first producer client and from the second producer-client a series of packets that include one or more captured client-based events generated in response to the instructions requesting capture of one or more client based events, the received series of packets including one or more timestamps associated with each of the client based events;

integrating a test pattern into the original series of packets to validate the timing accuracy of the captured client-based events and disregarding any packets send back to the server that failed the test pattern test;

merging separately the original series of packets with the second and third series of packets send back by the first producer-client and the second producer-client while deleting any redundant or unnecessary information to thereby form a merged series of packets and moving any series of packets within the merged series of packets as needed to preserve streaming performance;

streaming the new merged streams to one or more observer-clients;

wherein the instructions in the original series of packets include time stamped Remote Procedure Calls (RPC) that include time stamps, procedure identification with a request ID.

25. The method of claim 24 wherein the capture client based events in the second and third series of packets are deliverable to one or more third parties that are able to interactively modify capture requests and parameters in the original series of packets sent by the server and rerun in the entirety the streaming and receiving.

26. The method of claim 24 wherein the time stamped Remote Procedure Calls (RPC) include at least one parameter selected from the group including a client ID, session ID, one or more object IDs and one or more data buffers not identified by an ID.

27. The method of claim 24 wherein the client based events includes information reflective of one or more states of the first or second producer-clients.

28. The method of claim 24 further wherein streaming the original series of packets to the first and second producer-clients further includes streaming the original packets with an adaptive streaming technique.

29. The method of claim 24 wherein the merging further includes converting the merged series of packets to a video format that conforms to a standard video protocol.

30. The method of claim 24 wherein the server reformats or summarizes the sensor data sent back by the first and second producer-clients and sends the reformatted or summarized sensor data with the merged streams to the observer-clients.

31. The method of claim 24 wherein the server original streams dynamically turn on and off, on an as-needed basis, sensor devices on the first and second producer-clients.