MOBILE SEARCH-READY SMART DISPLAY TECHNOLOGY UTILIZING OPTIMIZED CONTENT FINGERPRINT CODING AND DELIVERY

• US 20170195717A1
• Filed: 12/30/2015
• Published: 07/06/2017
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: Zhang, Zhi, Wang, Haohong, Ning, Guanghan, Ren, Xiaobo
• Granted Patent: US 9,807,453 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A method for a mobile search-ready smart display system containing a smart display having a screen and at least one mobile device having a camera including extracting a plurality of video fingerprints from an ongoing content displayed on the screen. The ongoing content has video frames corresponding to the video fingerprints; determining one or more representative video fingerprints of the video frames from the plurality of video fingerprints; generating one or more encoded representative video fingerprints through encoding the one or more representative video fingerprints into a pixel translucency change; and delivering the one or more encoded representative video fingerprints via a screen-camera communication. The method also includes optimizing the encoding of the one or more representative video fingerprints and the delivering of the one or more encoded representative video fingerprints to find a balance between an error resilience and transmission time.

Claims

1. A method for a mobile search-ready smart display system containing a smart display having a screen and at least one mobile device having a camera, the method comprising:

extracting, by a hardware processor, a plurality of video fingerprints from an ongoing content displayed on the screen, the ongoing content having video frames corresponding to the video fingerprints;

determining, by the hardware processor, one or more representative video fingerprints of the video frames from the plurality of video fingerprints;

generating, by the hardware processor, one or more encoded representative video fingerprints through encoding the one or more representative video fingerprints into a pixel translucency change;

delivering, by the hardware processor, the one or more encoded representative video fingerprints via a screen-camera communication; and

optimizing, by the hardware processor, the encoding of the one or more representative video fingerprints and the delivering of the one or more encoded representative video fingerprints by solving a polynomial time dynamic problem to find a balance between an error resilience and a transmission time, wherein the balance between the error resilience and the transmission time indicates a maximum video retrieval accuracy.

2. The mobile search-ready smart display method according to claim 1, further comprising:

decoding, by the hardware processor, the one or more encoded representative video fingerprints to uncoded video fingerprints;

searching, by the hardware processor, for a matching video having video frames corresponding to the uncoded video fingerprints in a content database; and

returning, by the hardware processor, searching results to a user.

3. The mobile search-ready smart display method according to claim 1, further comprising:

solving, by the hardware processor, the polynomial time dynamic problem using a Lagrange multiplier method.

4. The mobile search-ready smart display method according to claim 1, wherein:

the pixel translucency change is an alpha value change.

5. The mobile search-ready smart display method according to claim 4, wherein:

the alpha value change of each pixel is based on color distribution of the video frames and frame transition.

6. The mobile search-ready smart display method according to claim 1, wherein:

the smart display is an online display or an offline display.

7. The mobile search-ready smart display method according to claim 1, wherein optimizing the encoding of the representative video fingerprints and the delivering of the encoded representative video fingerprints further includes:

finding, by the hardware processor, a maximum expected value of a successful probability that the fingerprint is able to be transmitted when given limited bits are used by the fingerprint.

8. The mobile search-ready smart display method according to claim 7, wherein:

finding the maximum expected value of the successful probability is represented by finding;

9. A non-transitory computer-readable medium having computer program for, when being executed by a hardware processor, performing a content retrieval method, the method comprising:

extracting, by the hardware processor, a plurality of video fingerprints from an ongoing content displayed on the screen, the ongoing content having video frames corresponding to the video fingerprints;

determining, by the hardware processor, one or more representative video fingerprints of the video frames from the plurality of video fingerprints;

generating, by the hardware processor, one or more encoded representative video fingerprints through encoding the one or more representative video fingerprints into a pixel translucency change;

delivering, by the hardware processor, the one or more encoded representative video fingerprints via a screen-camera communication; and

optimizing, by the hardware processor, the encoding of the one or more representative video fingerprints and the delivering of the one or more encoded representative video fingerprints by solving a polynomial time dynamic problem to find a balance between an error resilience and a transmission time, wherein the balance between the error resilience and the transmission time indicates a maximum video retrieval accuracy.

10. The non-transitory computer-readable medium according to claim 9, the method further comprising:

decoding, by the hardware processor, the one or more encoded representative video fingerprints to uncoded video fingerprints;

searching, by the hardware processor, for a matching video having video frames corresponding to the uncoded video fingerprints in a content database; and

returning, by the hardware processor, searching results to a user.

11. The non-transitory computer-readable medium according to claim 9, the method further comprising:

solving, by the hardware processor, the polynomial time dynamic problem using a Lagrange multiplier method.

12. The non-transitory computer-readable medium according to claim 9, wherein:

the pixel translucency change is an alpha value change.

13. The non-transitory computer-readable medium according to claim 12, wherein:

the alpha value change of each pixel is based on color distribution of the video frames and frame transition.

14. The non-transitory computer-readable medium according to claim 9, wherein:

the smart display is an online display or an offline display.

15. The non-transitory computer-readable medium according to claim 9, wherein optimizing the encoding of the representative video fingerprints and the delivering of the encoded representative video fingerprints further includes:

finding, by the hardware processor, a maximum expected value of a successful probability that the fingerprint is able to be transmitted when given limited bits are used by the fingerprint.

16. The non-transitory computer-readable medium according to claim 15, wherein:

finding the maximum expected value of the successful probability is represented by finding;

17. A method for a mobile device having a camera, comprising:

receiving, by a hardware processor, one or more encoded representative video fingerprints of a video frame delivered via a screen-camera communication via a screen-camera communication;

decoding, by the hardware processor, the one or more encoded representative video fingerprints to uncoded video fingerprints;

searching, by the hardware processor, for a matching video having video frames corresponding to the uncoded video fingerprints in a content database; and

returning, by the hardware processor, searching results to a user;

wherein;

the one or more encoded representative video fingerprints of the video frame are generated by encoding one or more representative video fingerprints of the video frame into a pixel translucency change,

the one or more representative video fingerprints are selected from a plurality of video fingerprints extracted from an ongoing content displayed on a screen of a smart display,

the encoding of the representative fingerprints and the delivering of the encoded representative video fingerprints are optimized by solving a polynomial time dynamic problem to find a balance between an error resilience and a transmission time, and

the balance between the error resilience and the transmission time indicates a maximum video retrieval accuracy.

18. The method for a mobile device having a camera according to claim 17, further comprising:

solving, by the hardware processor, the polynomial time dynamic problem using a Lagrange multiplier method.

19. The method for a mobile device having a camera according to claim 17, wherein:

the encoding of the representative fingerprints and the delivering of the encoded representative video fingerprints are optimized through finding a maximum expected value of a successful probability that the fingerprint is able to be transmitted when given limited bits are used by the fingerprint.

20. The method for a mobile device having a camera according to claim 19, wherein:

finding the maximum expected value of the successful probability is found through represented by finding;