Video image distribution method

• US 9,525,892 B2
• Filed: 05/27/2014
• Issued: 12/20/2016
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Nextdrive Holdings Company Limited
• Original Assignee: Nextdrive, Inc.
• Inventors: Lin, Shu-Yu, Yen, Che-Yuan, Chen, Chung-Kuang
• Primary Examiner(s): Williams, Jeffery
• Application Number:
  US14/288,144
• Publication Number:
  US 20150350518A1
• Time in Patent Office: 938 Days
• Field of Search: 348/61
• US Class Current: 1/1
• CPC Class Codes: H04N 21/2183 Cache memory caches in web servers or browsers G06F16/9574; intermediate storage and caching in data networks H04L67/568
• CPC Class Codes: H04N 21/239 Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/4222 Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy remote control device for a television receiver integrated into a mobile phone H04M1/72415
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/4431 characterized by the use of Application Program Interface [API] libraries
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/8545 for generating interactive applications
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot

Abstract

A video image distribution method used in a mobile electronic device is disclosed to include the step of controlling the video camera and starting image capture execution command, the step of temporarily storing original video camera images, the step of enabling the proprietary camera layer to use resource images for the creation of an image processing chain, the step of allocating hardware and software resources and constructing an image processing node Then enabling image processing chain to add source images to multiple processed images for storage, and the step of creating an application user linked service execution command and distributing temporarily stored multiple images to multiple application users. As long as the hardware and software resources permit, the method of the invention can serve multiple application users to meet different needs of different application users for different format settings, achieving optimal results, and facilitating operation of the mobile electronic device.

Claims

1. A video image distribution method used in a mobile electronic device comprising a hardware platform, an operating system installed in said hardware platform and an image capture interface, said operating system having built therein a proprietary camera layer and a driver, said driver being controlled to receive video images from a video camera through said image capture interface and to transmit received video images to said proprietary camera layer for dynamically creating image processing chains and multiple images queues for distribution to multiple application users, the video image distribution method comprising:

(a01) controlling said video camera and starting image capture execution command;

(a02) temporarily storing original video camera images;

(a03) said proprietary camera layer using source images to create image processing chain;

(a04) allocating hardware and software resources and constructing an image processing node, and then enabling image processing chain to add source images to multiple processed images for storage;

(a05) creating an application user linked service execution command and distributing temporarily stored multiple images to multiple application users.

2. The video image distribution method as claimed in claim 1, wherein step (a01) of controlling said video camera and starting image capture execution command uses the hardware of said video camera, said driver and/or said image capture framework to control the operation of said video camera.

3. The video image distribution method as claimed in claim 1, wherein step (a03) of creating image processing chain comprises the sub steps of:

(b01) a camera API (application programming interface) of said operating system accepting request for parameter settings from one said application user;

(b02) determining whether there is an image processing chain of the parameter setting requested by the application user, and then proceeding to step (b03) if yes, or step (b05) if not;

(b03) creating a service execution command and linking said service execution command to the image processing chain and the application user;

(b04) completing the request of the application user;

(b05) allocating the hardware and software resources, and then searching a suitable temporarily stored source image and creating an image processing node;

(b06) determining whether or not the hardware and software resource allocation and image processing node creation are succeeded, and then, proceeding to step (b07) if succeeded, or step (b09) if not succeeded;

(b07) determining whether the request for parameter setting from the application user is satisfied, and then proceeding to step (b08) if satisfied, or step (b09) if not satisfied;

(b08) linking the image processing node and creating a new image processing chain, and then returning to step (b03); and

(b09) refusing the request from the application user.

4. The video image distribution method as claimed in claim 3, wherein when one application user makes a video camera parameter query, the procedure comprises the steps of:

(c01) the application user starting video camera parameter query through an application programming interface, a video camera service and said proprietary camera layer;

(c02) determining whether there is an image processing chain and adequate hardware resources, and then proceeding to step (c04) if yes, or step (c03) if not;

(c03) inputting a list of commonly used supported image formats;

enabling the hardware resource manager to input the record of use of hardware resources, the processing of hardware supported image format and the image processing list in action, and then proceeding to step (c04);

(c04) generating supported video camera parameters subject to the hardware resources and the existed image processing chain, and then sending the generated supported video camera parameters through said proprietary camera layer and the respective video camera service and said application program interface to the application user.

5. The video image distribution method as claimed in claim 3, wherein when one application user starts to set video camera parameters, the video camera parameter setting procedure comprises the steps of:

(d01) the application user setting video camera parameters through the respective video camera service, said application program interface and said proprietary camera layer;

(d02) checking whether there is an image processing chain, and then, proceeding to step (d03) if yes, or step (d04) if not;

(d03) creating a linked data corresponding to the video camera user and the respective image processing chain, and then sending a message of SETTING SUCCEEDED back to the application user via the respective video camera service, said application program interface and said proprietary camera layer;

(d04) checking whether there is an adequate format for hardware resource processing, and then proceeding to step (d05) if yes, or step (d06) if not;

(d05) generating an image processing chain subject to the requested format, and then returning to step (d03);

(d06) refusing parameter setting, and then sending a message of SETTING FAILED back to the application user via the respective video camera service, said application program interface and said proprietary camera layer.

6. The video image distribution method as claimed in claim 1, wherein in said step (a04), the procedure of enabling image processing chain comprises the sub steps of:

(e01) the application user starting up image capture through said application program interface, said video camera service and said proprietary camera layer;

(e02) getting the image processing chain linked data of said video camera corresponding to this application user;

(e03) checking whether the image processing chain is created, and then, proceeding to step (e04) if created, or step (e09) if not created;

(e04) checking whether the image processing chain is enabled, and then, proceeding to step (e05) if enabled, or step (e06) if not enabled;

(e05) creating a service execution command for this application user, and then sending a message of ENABLE SUCCEEDED back to the application user via said video camera service, said application program interface and said proprietary camera layer;

(e06) enabling the image processing chain;

(e07) checking whether the image node involved hardware is enabled, and then, returning to step (e05) if enabled, or proceeding to step (e08) if not enabled;

(e08) checking whether enabling and setting of the related hardware is succeeded, and then, returning to step (e05) if succeeded, or proceed to step (e09) if not succeeded; and

(e09) enable failed;

sending a message of ENABLE FAILED back to the application user via said video camera service, said application program interface and said proprietary camera layer.

7. The video image distribution method as claimed in claim 1, wherein in said step (a04), the processing of the image processing chain comprises the sub steps of:

(f01) noticing that there are temporarily stored source images;

(f02) determining whether or not the source images contain a new image, and then proceeding to step (f03) if yes, or step (f06) if not;

(f03) processing the source images using the assigned parameters and software and hardware resources;

(f04) determining whether or not the image processing is succeeded, and then, proceeding to step (f05) if yes, or step (f06) if not;

(f05) adding the processed image to the assigned output location for storage and then proceeding to step (f06); and

(f06) waiting for a next registered source image notice.

8. The video image distribution method as claimed in claim 1, wherein after the procedure of distributing temporarily stored multiple images to said application users in said step (a05), the procedure of disabling the image processing chain comprises the sub steps of:

(g01) the application user stopping image capture through the video camera, the application program interface, the video camera service and the proprietary camera layer;

(g02) getting the processing chain linked data from the video camera corresponding to this application user;

(g03) checking whether or not the image processing chain has been created, and then, proceeding to step (g04) if yes, or step (g08) if not;

(g04) checking whether or not the image processing chain has been enabled, and then, proceeding to step (g05) if yes, or step (g08) if not;

(g05) disabling the image processing chain;

(g06) checking whether or not the related image node has been shared by other image processing chain, and then, proceeding to step (g08) if yes, or step (g07) if not;

(g07) disabling the image node associated hardware and releasing the related resources, and then, proceeding to step (g08);

(g08) disabling the service execution command, and then, sending a message of DISABLE SUCCEEDED back to the application user via the proprietary camera layer, the respective video camera service and the application program interface.

9. The video image distribution method as claimed in claim 1, wherein said hardware platform of said mobile electronic device comprises a CPU, a HW/SW shared memory, a HW camera interface (DMA), an image sensor+ISP (image signal processor), a hardware video encoder, a 2D processor (scaler, color space converter, blitter), and a 3D processor (pixel shader, fragment shader).

10. The video image distribution method as claimed in claim 1, wherein said operating system is selected from the group of Android operating system, iOS operating system and Windows Phone operating system.

11. The video image distribution method as claimed in claim 1, wherein Android operating system comprises Android camera API, Android APIs and Android Core Services/System Frameworks.

12. The video image distribution method as claimed in claim 1, wherein said proprietary camera layer comprises Camera Recorder, Streaming Server, Buffer Queue, Image Processing and Original Camera Buffer Queue.

13. The video image distribution method as claimed in claim 1, wherein said proprietary camera layer is a soft ware for running an algorithm on a host processor.

14. The video image distribution method as claimed in claim 1, wherein said proprietary camera layer is a hardware selected from the group of video encoder, post-processor and 2D accelerator.

15. The video image distribution method as claimed in claim 1, wherein said proprietary camera layer is a pixel shader or fragment shader for running a video processing algorism on a GPU (graphic processing unit).

16. The video image distribution method as claimed in claim 1, wherein said driver is Linux kernel comprising Camera Driver/V4L2 Framework, Android IPC Driver and Android ION Buffer Management Driver.

17. The video image distribution method as claimed in claim 1, wherein said application users of said mobile electronic device are Skype, Line, WeChat, Viber or Yahoo Messenger application programming interfaces.