Method and apparatus for video streaming

• US 9,001,216 B2
• Filed: 10/12/2012
• Issued: 04/07/2015
• Est. Priority Date: 10/12/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sampathkumaran, Sriram, Neumann, Helmut, Yam, Eric, Hong, Seungwook, Agnihotri, Tanmay
• Primary Examiner(s): DIEP, TRUNG T
• Application Number:
  US13/651,290
• Publication Number:
  US 20140104440A1
• Time in Patent Office: 907 Days
• Field of Search: 348/42, 348/207.1, 3482111-2113, 34821111-21112, 348/211.99, 348/744
• US Class Current: 348/207.1
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4415 using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning cryptography using biological data H04L9/3231; authentication in networks using biometric H04L63/0861
• CPC Class Codes: H04N 21/4431 characterized by the use of Application Program Interface [API] libraries

Abstract

An apparatus and method are provided for a client application to provide a video stream as well as data regarding the video stream to an output device, where the data is generated from a process performed at a layer lower than the client application layer. By this, multiple client applications can simultaneously utilize the data generated by the lower-level process while at least one client application can effect streaming of the video stream to an output device.

Claims

1. A method comprising:

receiving at an arbitration module of a computing device, wherein the arbitration module resides on a native layer of the computing device, from at least one client application at least a request for a video stream, a request for results of at least one process performed on the video stream, and at least one location information element for display of the video stream on a video plane;

receiving of the video stream by the arbitration module;

performing by the arbitration module the at least one process on the video stream, thereby generating the results of the at least one process;

outputting from the arbitration module the video stream, the results of the at least one process, and the at least one location information element for display of the video stream on the video plane;

receiving by a video pipeline element coupled to the arbitration module at least the video stream and the at least one location information element for display of the video stream on the video plane;

streaming by the video pipeline element at least the video stream to the video plane in accordance with the at least one location information element;

providing by the client application access of the video plane to an output device, whereby the video plane is displayed on the output device;

generating by the client application a graphical layer, wherein the graphical layer includes at least one element of visual information generated from at least the results of the at least one process; and

outputting of the graphical layer from the client application to the output device, wherein the graphical layer and the video plane are simultaneously displayed on a display of the output device.

2. The method of claim 1 wherein performing by the arbitration module the at least one process on the video stream further comprises adding at least one visual representation of the results to the video stream prior to outputting the video stream from the arbitration module, and wherein outputting from the arbitration module the results of the at least one process further comprises outputting the video stream with the added at least one visual representation of the results.

3. The method of claim 1 further comprising:

receiving at the at least one client application from the arbitration module the at least one result of the at least one process;

and wherein generating by the client application the graphical layer further comprises generating by the at least one client application at least one visual representation of the at least one result of the at least one process performed on the video stream and providing the at least one visual representation of the at least one result of the at least one process performed on the video stream to the output device.

4. The method of claim 1 wherein the at least one location information element includes a location parameter in relation to the display of the output device and a size parameter in relation to the display of the output device, and wherein the graphical layer includes a void corresponding to the location parameter and the size parameter.

5. The method of claim 1 wherein the video pipeline element comprises a gStreamer-compatible video pipeline element in a gStreamer-compatible video pipeline.

6. The method of claim 1 wherein the video pipeline element comprises an OpenMAX™

-compatible video pipeline element in an OpenMAX™

-compatible video pipeline.

7. The method of claim 1 wherein the computing device utilizes an Android™

-compatible operating system.

8. The method of claim 1 wherein the arbitration module further comprises a natural inputs engine, and wherein performing by the arbitration module of the at least one process on the video stream to generate the at least one result of the at least one process further comprises determining at least one natural input present in the video stream.

9. The method of claim 8 wherein the natural inputs engine further comprises a face detection engine configured to determine at least one aspect relating to at least one face present in the video stream.

10. The method of claim 1 wherein receiving the video stream at the arbitration module further comprises receiving the video stream from a camera device.

21. The method of claim 1, wherein the at least one client application is a Java™

application.

11. An apparatus comprising:

an arbitration module of a computing device, wherein the arbitration module resides on a native layer of the computing device, configured to;

receive from at least one client application at least a request for a video stream, a request for at least one result of at least one process performed on the video stream, and at least one location information element for display of the video stream on a video plane;

receive the video stream;

perform the at least one process on the video stream thereby generating the at least one result of the at least one process; and

output the video stream, the at least one result of the at least one process, and the at least one location information element for display of the video stream on the video plane; and

a video pipeline element of the computing device coupled to the arbitration module, the video pipeline element configured to;

receive at least the video stream and the at least one location information element for display of the video stream on a video plane; and

stream at least the video stream to the video plane in accordance with the at least one location information element;

the at least one client application coupled to the arbitration module, the at least one client application configured to;

generate a graphical layer, wherein the graphical layer includes at least one element of visual information generated from the at least one result of the at least one process;

provide to an output device, wherein the output device is coupled to the client application and the video plane, the graphical layer; and

provide to the output device access to the video plane and access to the at least one result of the at least one process performed on the video stream, wherein the graphical layer and the video plane are simultaneously displayed on a display of the output device.

12. The apparatus of claim 11 further comprising:

at least one camera coupled to the arbitration module, the at least one camera configured to generate the video stream.

13. The apparatus of claim 11 further comprising:

an Android™

-compatible operating system in which the at least one client application operates.

14. The apparatus of claim 11 wherein the video pipeline element further comprises a gStreamer-compatible video pipeline element in a gStreamer-compatible video pipeline.

15. The apparatus of claim 11 wherein the video pipeline element further comprises an OpenMAX™

-compatible video pipeline element in an OpenMAX™

-compatible video pipeline.

16. The apparatus of claim 11 wherein the arbitration module is further configured to:

perform the at least one process on the video stream to generate the at least one result by adding at least one visual representation of the at least one result to the video stream prior to outputting the video stream; and

output the at least one result of the at least one process by outputting the video stream with the added at least one visual representation of the at least one result.

17. The apparatus of claim 11 wherein the at least one client application is further configured to:

receive from the arbitration module the at least one result of the at least one process;

generate additional visual information by generating at least one visual representation of the at least one result of the at least one process performed on the video stream; and

include the at least one visual representation in the graphical layer.

18. The apparatus of claim 11 wherein

the at least one location information element includes a location parameter in relation to the display of the output device and a size parameter in relation to the display of the output device, and wherein the graphical layer includes a void corresponding to the location parameter and the size parameter.

19. The apparatus of claim 11 wherein the arbitration module further comprises a natural inputs engine configured to perform the at least one process on the video stream to generate the at least one result of the at least one process by processing the video stream to determine at least one natural input present in the video stream.

20. The apparatus of claim 19 wherein the natural inputs engine further comprises a face detection engine configured to determine at least one aspect relating to at least one face present in the video stream.

22. The apparatus of claim 11, wherein the at least one client application is a Java™

application.