Machine-Based Object Recognition of Video Content
1. A method comprising:
- receiving, by an electronic device, a request from a user for video content;
requesting, by the electronic device, the video content from a video server;
requesting, by the electronic device, metadata about an object associated with a person appearing in the video content from a metadata server, respective pieces of the metadata being aligned with respective frames of the video content;
displaying, via a display of the electronic device, a first frame of the video content to the user, the first frame including an image of the person;
receiving, by the electronic device, a request from the user for the metadata about the object associated with the person; and
in response to the request from the user, displaying, via the display of the electronic device, a first piece of metadata about the object with on the video content, the first piece of metadata being aligned with the first frame.
Current interfaces for displaying information about items appearing in videos are obtrusive and counterintuitive. They also rely on annotations, or metadata tags, added by hand to the frames in the video, limiting their ability to display information about items in the videos. In contrast, examples of the systems disclosed here use neural networks to identify items appearing on- and off-screen in response to intuitive user voice queries, touchscreen taps, and/or cursor movements. These systems display information about the on- and off-screen items dynamically and unobtrusively to avoid disrupting the viewing experience.
- 1. A method comprising:
receiving, by an electronic device, a request from a user for video content; requesting, by the electronic device, the video content from a video server; requesting, by the electronic device, metadata about an object associated with a person appearing in the video content from a metadata server, respective pieces of the metadata being aligned with respective frames of the video content; displaying, via a display of the electronic device, a first frame of the video content to the user, the first frame including an image of the person; receiving, by the electronic device, a request from the user for the metadata about the object associated with the person; and in response to the request from the user, displaying, via the display of the electronic device, a first piece of metadata about the object with on the video content, the first piece of metadata being aligned with the first frame.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
- 12. A method comprising:
receiving, at a server, a first request from a user for metadata about an object appearing in a video on a user device; determining, by the server, that a metadata database does not have the metadata about the object; in response to determining that the metadata database does not have the metadata about the object, sending a query, by the server, for an identity of the object appearing in the video to an object recognition server; receiving, by the server, the identity of the object appearing in the video from the objection recognition server; obtaining, by the server, the metadata about the object based on the identity of the object; and providing, by the server, the metadata for display to the user, the user device displaying at least a portion of the metadata with the video.
- View Dependent Claims (13, 14, 15, 16, 17, 18)
The present application claims the benefit, under 35 U.S.C. § 120, as a continuation-in-part (CIP) of PCT Application No. PCT/US2017/062284, filed Nov. 17, 2017, entitled “Machine-Based Object Recognition of Video Content.”
PCT/US2017/062284 claims priority, under 35 U.S.C. § 119, to U.S. Application No. 62/423,495, entitled “Methods and Apparatus for Displaying Products Related to Video Content” and filed on Nov. 17, 2016.
Each of the above-mentioned applications is incorporated herein by reference in its entirety.
The state of video has remained largely unchanged for decades. While some attempts have been made to modernize video through the addition of interactivity, these attempts have fallen short for various reasons. For one, the information provided through previous attempts at interactivity has generally been extremely limited. This is largely because the manpower required for manually annotating a video including hundreds of thousands of frames is prohibitively expensive. In addition, previous interactive video interfaces have been unintuitive and very disruptive to the viewing experience.
Embodiments of the present technology address problems with annotating videos and displaying interactive video interfaces. First, a machine learning engine that can automatically generate metadata for media content reduces the manpower required for thorough annotation of content. Second, new interactive interfaces seamlessly relay this metadata to users without interrupting the users'"'"' goal of viewing content. These two elements combined allow for the broad distribution and adoption of detailed interactive video.
The present technology includes a method for identification of objects featured in video content and displaying user interfaces that enables users to retrieve more information about the identified objects. Examples of user interfaces that can be used individually or together with a real-time image recognition engine are discussed with reference to an Overlaid Video Player user interface and an Adjacent Layout user interface. Together, the Overlaid Video Player user interface and the Adjacent Layout user interface address a variety of technical problems associated with current technology for real-time image recognition, real-time retrieval of information of recognized objects and other suitable user-initiated processes related to objects recognized in video content. Although the examples of the subject technology are discussed in the context of user interactions related to browsing and buying items featured in video content, the subject technology could be employed in contexts such as surveillance, evidence gathering, and other suitable contexts.
For instance, current solutions for purchasing products related to video content are difficult to use and have a low purchase rate. To increase engagement and purchase click through rate, the Overlaid Video Player makes purchasing products through video content more intuitive for users via an interface that displays product images in real-time alongside the video. It presents the products on top of the video in a non-intrusive way that allows users to easily click to purchase in a minimal number of steps. The Adjacent Layout can also bring more visibility to the products in video content and are another opportunity for users to make product purchases.
Other technical problems include updating the Overlaid Video Player'"'"'s interface often enough to provide a seamless display. In order to maintain context with the video, the Overlaid Video Player interface should be updated frequently (e.g., at a frequency of about 1-4 times per second). This rate is too rapid to execute a network request for each interval.
To address this problem, the Overlaid Video Player downloads key data for each video segment (e.g., 0-15 minutes of video content) at the beginning of the segment in a compressed format and caches the key data on the user'"'"'s device. This allows the interface to be updated frequently by using the cached data on the device, which is much faster than performing an external network request to fetch the data.
Compatibility is another technical consideration for the Overlaid Video Player user interface. The Overlaid Video Player user interface should interface with many different underlying video player technologies (e.g., HTML5, Flash, etc.) in order to be distributed broadly. This includes being able to play/pause/seek the video, and being informed about the state of the video (e.g., current time, playing/paused, whether an ad is playing, etc.). To address this problem, the Overlaid Video Player interface may be coupled to a generic relay (
Embodiments of the present technology include various methods and systems. Some of these methods include receiving, by an electronic device, a request from a user for video content; requesting, by the electronic device, the video content from a video server; and requesting, by the electronic device, metadata about an object associated with a person appearing in the video content from a metadata server. Respective pieces of the metadata are aligned with respective frames of the video content. The electronic device shows, on its display, a first frame of the video content to the user. This first frame includes an image of the person and may or may not show the object associated with the person. In response, the electronic device receives a request from the user for the metadata about the object associated with the person. And in response to the request from the user, the electronic device displays a first piece of metadata about the object with the video content. This first piece of metadata may be aligned with the first frame.
In some examples of these methods, receiving the metadata includes receiving product identification information for objects worn by every person appearing in the video content. This metadata may include links for obtaining the objects worn by every person appearing in the video content. The electronic device may display a corresponding link for obtaining the object, and the corresponding link may dynamically redirect the user to a website for obtaining the object. In some cases, the metadata includes a link for obtaining an object similar to the object worn by the person appearing in the video content. The electronic device may display this type of link as well.
The electronic device may receive the user'"'"'s request in any of a variety of formats. For instance, a microphone of the electronic device may receive the user'"'"'s request as a voice command or voice query. Or the electronic device may receive the user'"'"'s request by detecting, proximate to a play bar area of the display of the electronic device, a cursor movement or touch on a touchscreen.
In some examples of these methods, the electronic device stores the metadata about the object in a memory of the electronic device before displaying the frame of the video content to the user. The electronic device may also request metadata about another object associated with another person appearing in a second frame of the video content while displaying the video content to the user. And the electronic device may pause playback of the video content in response to the request from the user.
Other embodiment methods include receiving, at a server, a first request from a user for metadata about an object appearing in a video on a user device and determining, by the server, that a metadata database does not have the metadata about the object. In response to determining that the metadata database does not have the metadata about the object, the server sends a query for an identity of the object appearing in the video to an object recognition server. The server receives the identity of the object appearing in the video from the objection recognition server and obtains the metadata about the object based on the identity of the object. The server then provides the metadata for display to the user, and the user device displays at least a portion of the metadata with the video. If desired, this metadata is stored in the metadata database and retrieved in response to subsequent requests for the metadata.
In some case, the objection recognition server implements at least one neural network that recognizes the object appearing in the video. The neural network may do this by generating an embedding representing the object appearing in the video and performing a comparison of this embedding to a plurality of embeddings stored in an object database. Respective embeddings of the plurality of embeddings represent respective objects of a plurality of objects. The neural network may determine the identity of the object based on the comparison, e.g., by identifying a closest match from among the plurality of objects as the object. If desired, the neural network may identify, based on the comparison, another object similar to the original object from among the plurality of objects. In these cases, the server may provide metadata associated with this other object to the user. And the user device may display at least a portion of the metadata associated with the other object to the user.
It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.
The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and/or structurally similar elements).
An overall view of the system can be seen in
On the device and video player platform (in this example, a PC with a web browser), the user navigates to a page where a video can be played. The website requests the video at 109 from the Video Server 111, and the Video Server responds to the request by transmitting video data (not shown in
The code to integrate the interactive components on the video player platform can be added to the Content Provider'"'"'s code that displays the video or the page that the video appears on. Alternatively, the code can be developed and served to the video player platform by a Third Party Metadata Service which provides the code (or an application programming interface (API)) for the Content Provider to integrate on the video player platform. Another alternative is for the Content Provider or a third party to place the code in a plug-in or extension (e.g., a web browser extension or a streaming media player plugin) that can detect content and inject the interactive component code onto the page (even if the content or page is not owned by the Content Provider or third party that is injecting the code). In all cases, when the interactive components are integrated, the code retrieves the appropriate product images and information at 113 from the metadata server 115 (hosted by the Metadata Provider) and displays (e.g., at 117) them on screen for the user to see.
The interactive components (and metadata server) are not restricted to displaying products related to the video. They can be used to display information about anything related to the video, such as services, physical locations or general information about the video. For example, they can be used to display details about a restaurant that appears in the video and allow users to book a reservation to dine there. Or they can be used to display trivia or facts about the characters or how the video was made. The interactive components are typically associated to a range of frames and a point in time during the video, but can also be general and associated with the video as a whole. When referring to a “product” in this document, keep in mind that this also includes services, physical locations, or any information related to people or objects appearing in the video.
A user interacting with the interactive components may have the ability to see additional information about a product or purchase a product. In some cases, the product can be purchased directly from the Content Provider as shown at 121. For example, if the Content Provider is a product retailer, they may want to display a video that features their products. The user can then add the products to a shopping cart and purchase directly from the Content Provider/Retailer website where the video appears.
In another case, the Content Provider is not a retailer and does not sell their own products. An example of this would be a television studio'"'"'s website, where they allow users to view (and shop) clips and full episodes of their shows. In this case, the Content Provider may choose to send users to one or more third party retailer web sites to complete their purchase as shown at 123. Another option for this case is to use a universal shopping cart solution or a retailer'"'"'s API that, when integrated, allows the user to make a purchase (or add to a retailer website'"'"'s shopping cart) without leaving the Content Provider'"'"'s website.
On televisions, media streamers or other devices and video platforms that do not allow for easy user input of payment (e.g., credit card) information, another integration option is possible. When the user indicates they want to purchase an item, the Content Provider'"'"'s (or the Third Party Video Service'"'"'s) code can display on the page a short URL and/or offer the option to receive an SMS message (with an URL) on their mobile device, where they can complete the purchase.
In response to receiving the user'"'"'s request, the video player 107 requests a video from the video server 111 (step 2). The video server 111 delivers the video to the video player 107 (step 3). Once the video starts playing at the video player 107, a video event shim 203 receives an event message (video frame time stamp) from the video player 107 indicating that playback has started (step 4). In addition, the video event shim 203 sends a notification message to a metadata controller 205 indicating that the playback of the requested video has started (step 5). The metadata controller 205 requests metadata associated with the corresponding event (video frame time stamp) from product server 207 (step 6), possibly in response to a user request via the interfaces described below, and the product server 207 locates, retrieves, and sends the requested metadata to the metadata controller 205 (step 7). Likewise, the metadata controller 205 sends the requested metadata to a metadata overlay 201 (step 8) for display by the video player 107.
If the user 203 selects on a product displayed by the metadata overlay 201 (step 9), the metadata overlay 201 sends a notification message to the metadata controller 205 (step 10). In response, the metadata controller 205 sends a command to the video event shim 203 to pause the video at the video player 107 (step 11) and the video event shim sends a pause message to the video player 107 (step 12). The metadata controller 205 may also update the video overlay or adjacent layout to show detailed information for the product or takes an action such as opening a new browser tab with the retail page for the product.
The interactions illustrated in
The metadata is time-aligned to the video frames using the video frame time stamps. This enables the metadata controller 205 to request and queue metadata based on the state of the video playback, e.g., for real-time metadata requests or requests for metadata at intervals. For example, the video frames may be tagged with metadata at a rate of 1 to 4 frames per second (fps). For playback speeds of 24 fps, this works out to one to four frames tagged with metadata being displayed every second; for faster playback speeds (e.g., the 60 fps used in high-definition video), the tagged frame rate may remain at 1-4 fps, with a lower percentage of tagged frames.
The metadata for each tagged video frame may include object or product identification information (e.g., product IDs) for all objects and products associated with the people appearing in the video frame, regardless of whether the objects and products appear in the frame. In other words, the metadata may include object or product IDs for objects or products that don'"'"'t appear in the frame. For instance, even if a person'"'"'s feet or lower body don'"'"'t appear in the video frame, the metadata may include information about the shoes and pants worn by the person. This enables the system to respond to user queries like “What is he wearing?” by displaying information about every item worn by the person instead of just the items appearing on screen.
In some cases, the metadata controller 205 may temporarily store or query information about recent frames, e.g., frames displayed over the past few seconds or minutes. Again, this enables the system to respond to user queries like “What is he wearing?” or “What car was she driving?” by retrieving and displaying information about items that don'"'"'t appear on screen when the user makes his or her query.
If desired, the metadata may include information about items related to the items associated with the person or people appearing in the video. For instance, if one item is a particular leather jacket, the metadata may include information about similar leather jackets or about accessories that pair well with the leather jacket (e.g., boots or bags). Similar items may be linked by their product IDs to reduce the amount of data transferred among servers and controllers during the query process. The similar items may be identified before or during playback. If they are identified during playback, they can be identified according to predetermined relationships stored in a database or dynamically using one or more neural networks implemented using an object recognition server as described below.
Item metadata may also include links (e.g., uniform resource locators (URLs)) to websites that offer the items for sale. These links may be static and can be assigned ahead of time or as the metadata is requested, e.g., in response to a real-time internet auction like those conducted for placing internet advertisements. The links may point to a redirect page that redirects the user to a third-party website based on item availability, price, relationship, or the result of a real-time internet auction.
Real-Time Object Recognition
The object recognition server 200 can include one or more graphical processing units (GPUs) 213, random access memory (RAM) 211, a network interface (not shown in
Images in a user-requested video can be recognized for a first time using the real-time recognition application 210 while a user watches a video. Concurrently the real-time recognition application 210 can store metadata of the recognized objects, including a reference timestamp corresponding to a time within the video duration, and other suitable information associated with the objects included in the video frames of a video. Accordingly if a second user requests to watch the same video, the real-time recognition application 210 does not have to analyze the video again. Thus, the product server 207 can retrieve the data from the product metadata database 209.
In operation, the product server 207 can obtain metadata in real-time using the object recognition server 200 and fetch previously indexed metadata from the product metadata database 209. To access real-time or previously indexed metadata, the product server 207 can send metadata requests to product metadata database 209 to retrieve metadata for the display of product information on, for example, metadata overlay 201 discussed with reference to
When operating in real-time recognition mode, the product server 207 sends a request to the object recognition server 200 when the product server 207 receives a message indicating that the requested metadata was not found at the product metadata database 209 (step 2A). The real time recognition application 210 responds to metadata requests received from product server 207 by generating the requested metadata, including metadata describing and/or identifying, for example, products, characters, or other suitable entities shown at a video frame displayed at the video player 107 (shown in
Neural Networks for an Object Recognition Server
A neural network or a mapping function may be configured to extract features from images (e.g., visual features) or any of the aforementioned inputs. In some implementations, the neural network may be configured to use convolutional layers. Each convolutional layer'"'"'s parameters comprise a set of learnable filters (or kernels), which have a small receptive field. During computation of the model, each filter is convolved across the input, producing an activation map of that filter. As a result, the network learns filters that activate when it detects some specific type of feature at some spatial position in the input. Convolutional layers are particularly useful for learning translation invariance of features found in images. With properly tuned convolutional layers, a feature is equally recognized in one region of the image as in another.
The neural network may be a deep neural network (i.e., a neural network with more than one hidden layer). Deep neural networks utilize more parameters than simple neural networks and are therefore more capable of learning complex features.
The neural network may use a regularization method like batch normalization to improve training speed and reduce overfitting of the model.
The neural network may use a rectified linear unit activation to compute each individual neuron'"'"'s activation. Other possible activation functions that may be used include the log-sigmoid and tan-sigmoid.
The neural network may be configured to use pooling layers. It may also be configured to use fully connected layers. Fully connected layers are layers which have neurons that are fully connected to the previous layer'"'"'s neurons. For ‘n’ neurons in the current fully connected layer and ‘m’ neurons in the previous layer, there are ‘n×m’ connections and parameter weights between the two layers.
In operation, the neural network 303 shown in
The object recognition server compares these embeddings to embeddings stored in a database (e.g., the product database) that is updated constantly as new items become available. For instance, it may compute a distance in a multi-dimensional space between a computed embedding, e.g., for a jacket, and the stored embeddings. The second implementation identifies the stored embedding that is closest to the computed embedding as a match. The object recognition server identifies the item (e.g., the jacket) corresponding to the matching stored embedding as the item (in this case, the jacket) and returns the associated product ID or similar information. In some cases, the item identified by the neural network is identical to the object appearing in the video; in other cases, the item identified by the neural network is the closest match (most similar) to the object appearing in the video from among the objects represented by the stored embeddings.
The second implementation of the neural network may also identify the stored embedding(s) with the next-shortest distance(s) as corresponding to similar item(s). If desired, object recognition server may identify these similar items (e.g., similar jacket) and return the associated product IDs or similar information as well for display to the user.
Neural Network Training, Testing, and Validation
A real-time image recognition engine can be implemented via a machine-learning model 303, such as a neural network implemented by CPU, GPU, or other processor. This machine-learning model 303 can be trained, tested, and validated to identify or recognize in real-time, products, items, characters, and other suitable entities displayed on a video. Likewise, the machine-learning model 303 can infer associations between items, characters, and other suitable entities displayed on the video. The machine-learning model 303 can take as input a single video frame or a video segment (video segment defined as a sequence of video frames), and identify a set of items shown in the video. The machine learning model 303 can further infer an association or relationship among the identified set of items, for instance, the machine learning model can infer that a subset of the identified items are worn or carried by a character shown in the video. Accordingly, such a subset can be grouped under a category associated with the character.
The data input into the machine learning model 303 includes images (e.g., 401A and 401B) either whole images or curated images (e.g., cropped images). The curated images can be produced by a human or an automated system. The original images can be sourced from any media content that requires metadata tagging. The model may also have additional metadata input into it to aid in its task, such as whether the image is of a man or woman, or other type of human or non-human entity.
Testing and Validation
The machine learning model is validated on data with the same format as the training and testing set, however it is only validated on data points not seen during training or testing. Additional, qualitative testing can be done to validate the model, whereby suggested results from the model are shown to human moderators to evaluate for correctness holistically.
Inference of Associations
When properly trained, the model 303 understands associations between products. E.g. These shoes are worn by the same person as this jacket. This can be accomplished through a combination of pre-processing of the images input to the network and post-processing of the output of the network. For example, before being input to the network, images can be segmented by first running another neural network that is trained to detect people. Each of these segmented regions can then be input into the machine learning model separately, and their outputs associated with each other. To associate these regions across frames, facial recognition can be employed to match regions to each other. In cases where faces aren'"'"'t visible, alternative heuristics can be used, such as evaluating spatiotemporal similarities between regions, and calculating similarity between regions using the output of the machine learning model.
The output of the machine-learning model is a set of bounding boxes (e.g., bounding box 501 shown in
Overlaid Video Player User Interface
The Overlaid Video Player provides a way for consumers of video content to know what products appear in a video in real time, as they watch the video. It can be implemented using the systems shown in
As an example, the Overlaid Video Player can be implemented on a television show'"'"'s website (e.g., video shown at 6A), where users are able to watch clips and full-length videos of their favorite shows. The television studio has added the code to enable the Overlaid Video Player, and the user arrives on a page with the player on it (
When the user moves their cursor into the video player area, the Product Inset 601 appears, overlaid on the video area or next to the video area (
The Product Inset displays product images in real-time to match a product appearing in the frame at the corresponding time in the video. To make this possible, prior to enabling the shopping feature, the relevant product information is added to a database in the Metadata server by the Metadata Provider. In the database, every frame of the video (or every frame at a set interval, e.g., every 15th frame) can be associated with the products (and corresponding product information, such as images, price, purchase link, etc.) that appear in the given frame. The timestamp for when the frame (and products) appears in the video is also kept in the database, so that the code can display the products in real-time. In this example, the t-shirt worn by the male character can be seen in the Product Inset. The Product Image does not have to be an actual image of the product; it can be anything to represent the product seen on screen (e.g., text, an icon, or a drawing). In this example, while the video is playing, only one product image is shown at a time in the Product Inset. If there is more than one product in the frame, the other products are stacked below the first product and partially hidden from view.
In this example, the Product Inset shows only one product that appears in the current video frame—when the video is playing and the product in the inset no longer appears in the current frame, it changes to a product that appears in the new frame. For example, if the male character in the white t-shirt no longer appears in the frame, the t-shirt may no longer appear in the Product Inset. The code recognizes which frame/what time the video player is playing, and can pull the appropriate frame'"'"'s product information from the database and display it on screen. Being that not every single frame in a video is added to the database (it could be every 12 frames, for example), the product shown in the product inset for a given frame can be the product information for the closest, previous, or next frame that has product information tied to it. If there are no products in the current frame, the Product Inset does not show any product images. Alternatively, the last product image shown can remain in the Product Inset until a new product appears in the frame and replaces the previous product—this would ensure there is always a product appearing in the Product Inset.
The user can interact to see the products in the frame. This can be done by hovering (on desktop devices) the cursor on the Product Inset 601, clicking/tapping on the video itself, or clicking on a dedicated button that enables shopping. The hidden stacked products are expanded and made visible (
The topmost product 608 (the t-shirt) expands to show the corresponding Product Details Container 603. The Product Details Container 603 displays details about that product, and can also include one or more alternate products. This information can be stored in the Metadata database. In this example, the t-shirt 608 is by J. Crew and costs $24.50, and there are two alternate t-shirt products shown, with different prices. Details for the product and alternate products can include: product image, brand name, product name, price/stock status, retailer name, designer name, product type or description. It may also include a label to denote whether the product is an exact or close match to the product shown in the video (in this example, the J. Crew t-shirt is an exact match). It may also include a call to action button to purchase or get more information about the product or alternate products, or to add to a virtual shopping cart (in this case, a button with a price on it is shown, for purchasing the product). The alternate product(s) can be a similar product (like the two similar t-shirts in the example), the same product from a different retailer, a related product or an arbitrary product.
In some examples, all of the product information data is added to the database in advance. Some parts of the data can be dynamically generated and added to the Metadata database. For example, image recognition can be used to identify products in the frame and provide the closest match, selected from another internal database of products or retailer APIs. The alternate products could also be retrieved from a third party API (or internal system) that selects alternate products automatically based on various properties (e.g., name, color, price, UPC number, etc.). Other attributes for the product and alternative products, such as pricing and in-stock data, can also be dynamically updated on the Metadata database.
When the user hovers/taps on another product image, any open Product Details Container will be hidden and the new corresponding Product Details Container becomes visible (e.g., at 607 in
If the expanded products extend past the bottom of the video player, they can scroll to show more. On desktop this can be accomplished using the scroll wheel on the mouse or trackpad, or by moving the cursor to the bottom of the video player, and the expanded products will move upwards. On mobile, the user can drag the product images up and down to see more. On both platforms, the interface could also have up and down arrows above and below the expanded products list that can be clicked/tapped to scroll.
The expanded products may also show products or outfits from previous frames below the products from the current frame. This is something that the Content Provider or Third Party Overlaid Video Service can choose to set in the code. This allows users to view previous outfits without having to seek backwards to that specific frame. This is useful when there are not many products in a video, but can get confusing when there are many.
In another variation for the Product Inset, rather than only showing one product image with the other products hidden from view, the Content Provider or Third Party Overlaid Video Service may set the code to expose more than one (or all) of the product images for the frame in the Product Inset. The Product Inset with more than one product image in it can be visible at all times during video playback, or can be made visible when pausing the video in the ways mentioned above.
Clicking/tapping on the elements of the Product Details Container can open a third party website to purchase or get more information about the product or alternate products (FIG. 6E). In this example, clicking on the $17.70 button sends the user to Amazon.com, where they can purchase the jeans.
The Product Details Container collapses when the user hovers away from it (on desktop) or taps outside of the container (on mobile), or clicks/taps on the Play button (if the video is currently paused) and then the products in the Product Inset collapse into a stack to only show one product (
The Overlaid Video Player interface can be adapted to still images or animated gifs as well—this could be a promotional or set image for a television show or movie, a photo of any person or celebrity, or a street fashion photo. In this example (
The Adjacent Layout is another interface that provides a way for consumers of video content to know what products appear in a video. Whereas the Overlaid Video Player interface is located inside the video player, the Adjacent Layout is typically located outside of the video player. It is a combination of still images and details about the products that appear in those images. The experience is different for desktop and mobile users. Like the Overlaid Video Player, the Adjacent Layout interface can be implemented using the systems shown in
As an example, the Adjacent Layout could be implemented on a television show'"'"'s website, where users are able to watch clips and full-length videos of their favorite shows. The television studio has added the code to enable the Adjacent Layout, and the user arrives on a page with the Adjacent Layout on it (
One or more Outfits are displayed on the page (
Hovering over a Product Image can invoke an overlay with a button to click for more information about that product (
The Product Details Box 713 (
The Product Details Box 713 may also include one or more alternate products e.g., 710A, 710B, and 710C. Details for the alternate products can include: product image, brand name, product name, price/stock status, retailer name or description. It may also include a call to action button to purchase or get more information about the alternate products, or to add to a virtual shopping cart. The alternate product(s) can be a similar product, the same product from a different retailer, a related product (e.g., 711) or an arbitrary product. In this example, three similar styles are shown as alternate products.
When the Product Details Box opens, the Outfit Image may remain visible, to provide context for the products in the outfit. It may also include images of the other products from the outfit. In this example, these are shown on the right side of the Product Details Box. The coat is faded out since it has been selected, and users can click on the other products from the outfit. Clicking on another product from the outfit replaces the product information (including alternate products) with the information related to the newly selected product (
Clicking on the elements in the Product Details Box can open a third party website to purchase or get more information about the product or alternate products (
In a more complex implementation of the Adjacent Layout, more than one outfit may be displayed on the same page (
An alternative design for the Adjacent Layout is typically used for mobile devices. As an example, the Adjacent Layout mobile experience could be implemented on a television show'"'"'s website or software application, where users are able to watch clips and full-length videos of their favorite shows. The television studio has added the code to enable the Adjacent Layout, and the user arrives on a page with the Adjacent Layout on it (
One or more Outfit Cards (
Tapping on a Product Image Button at 805 will replace the Outfit Image with the respective Product Details Card 806 (
The Product Details Card displays details about the selected product. It includes some or all of the following information: brand name, product name, retailer name, designer, product type, description, price/stock status and exact/close match label. It may also include a call to action button to purchase or get more information about the product, or to add to a virtual shopping cart. In this example, a button with the $239.00 price is included. It may also include one or more alternate products. Details for the alternate products can include: product image, brand name, product name, price/stock status, retailer name or description. It may also include a call to action button to purchase or get more information about the alternate products. The alternate product can be a similar product, the same product from a different retailer, a related product or an arbitrary product.
Clicking on the elements in the Product Details Box can open a third party website to purchase or get more information about the product or alternate products (
One or more Outfits may be displayed on the page (
General Integration (Desktop and Mobile)
The Adjacent Layout can be displayed on various pages. It can be displayed on the same page as an Overlaid Video Player—typically below the video. It can be displayed on the same page as a regular (non-) video. Or it can be displayed on a page without a video player.
The Adjacent Layout can also be integrated within other types of Video Players. One or more Outfit can be displayed within the video player, at a specific time during the video—typically before or after a video advertisement, or at the end of the video (
Another version of the Adjacent Layout integrated within the video player is Dual View Shopping (
Interface for Desktop and Mobile Applications
Smart Television Interface with Voice Control
A user can watch a video (
The image 1103 of a displayed product or item can be enlarged (
While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein, and each of such variations and/or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the inventive teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.
The above-described embodiments can be implemented in any of numerous ways. For example, embodiments of designing and making the technology disclosed herein may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.
Further, it should be appreciated that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.
Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output (e.g., for viewing-impaired users). Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format.
Such computers may be interconnected by one or more networks in any suitable form, including a local area network or a wide area network, such as an enterprise network, and intelligent network (IN) or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.
The various methods or processes (e.g., of designing and making the technology disclosed above) outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.
In this respect, various inventive concepts may be embodied as a computer readable storage medium (or multiple computer readable storage media) (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other non-transitory medium or tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the invention discussed above. The computer readable medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present invention as discussed above.
The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present invention.
Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.
Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
The indefinite articles “a” and “an,” as used herein in the specification, unless clearly indicated to the contrary, should be understood to mean “at least one.”
The phrase “and/or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and/or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and/or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.
As used herein in the specification, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items in a list, “or” or “and/or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of” “only one of” or “exactly one of” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.
As used herein in the specification, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and/or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.
In the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to.