How the start-up jay makes metadata from film production usable

Sometimes you discover a pair of sneakers in a movie scene, that pretty red dress worn by the actress or a piece of furniture that you would like to buy. Or you wonder where those impressive cliffs by the sea are that are constantly visible in the background. And what was the name of the actor playing the supporting role again?
The production of films, series and documentaries also generates a lot of information about film locations, protagonists, costumes and props, but this has hardly been used or displayed to date. If you’re looking for the red dress, you google it and use the typical search terms on shopping platforms. The actors can be looked up on Wikipedia and IMDb using the second screen on a cell phone and no further information on filming locations can be found. Yet all this data exists and is collected for the production of a film – but then goes to waste because there is no central, digital tool to store it.

One tech solution for all film data

The Potsdam-based company transfermedia has now launched jay, a metadata technology that offers film productions and distributors an automated solution for recording and providing data. Whether data on film locations, props, actors or costumes: everything is linked and stored on a scene-by-scene basis thanks to metadata software. Viewers do not have to use a second screen to access the information, but can have everything displayed in the film – and even buy the displayed products directly with integrated in-stream shopping.

Streaming customers are familiar with part of the principle from Amazon X-Ray. Here, you can display the actors who are currently in the respective scene on the side of the screen. However, this is the result of a complex process in which scenes are manually subdivided and information about the protagonists is added afterwards. In addition, Amazon only offers information on the cast or soundtrack. With jay, all of a film’s metadata can now be captured during production for the first time – in an automated workflow. Distributors then receive the film directly with the linked data.

From infotainment to in-stream shopping, this opens up new business models for broadcasters and video-on-demand platforms. “One of the biggest challenges is to help initiate change and show what opportunities we have,” says Irmela Wrogemann, who speaks to broadcasters as CMO at transfermedia. “It’s all about change management, i.e. how to think, plan and play out content in the future. And also about opening up structures. Editing, production, distribution and technology have few interfaces in the world of broadcasters. If we work together in a more networked way, we can gain a lot. And ultimately offer the audience a better content experience.”

transfermedia has been conducting research in the field of metadata for some time now. The team around Peter Effenberg draws on many years of experience in film production, technical expertise and intensive research on the subject of metadata.

In-stream shopping: buy directly instead of spending time googling

We are familiar with the principle of shoppable content from social media. jay is now also making in-stream shopping possible for the film industry. This benefits the streaming market, as services such as Netflix, Disney and others are looking for new business models. Subscription figures are beginning to fall and the principle of constantly producing new content is reaching its limits. Some providers are already thinking about placing advertisements to generate new sources of income. This is where jay comes in with the option of in-stream shopping. Product placement, as regularly seen in major blockbusters from James Bond to Spider Man, works in a similar way – but in-stream shopping is much more straightforward, direct and measurable. Viewers do not have to research the manufacturer or brand afterwards; instead, they can be offered the right product with just one click while streaming. The basis for this is digital metadata capture with jay during production. In the near future, there are likely to be new jobs on film sets that involve data capture, and a data wrangler may then become part of the standard workforce. This is because it is worthwhile to completely merge the data from the screenplay or script continuity during the ongoing process. Data recognition using artificial intelligence is not currently an option – nor will it be in the near future. A red dress, for example, is easy to mix up and the algorithm is still overwhelmed and prone to error.

Understanding the needs of viewers

jay not only enables the central operation and storage of data, but also analyzes and verifies it using the DataQuality feature in order to adapt the information to the distributors’ standards. The broadcasters and streaming services are responsible for integrating it correctly into the respective platform. However, jay is on hand to advise them on implementation and is currently conducting user research to better understand the needs of viewers: What additional information are they particularly interested in? How do they want to interact with the data? And what is important to them in terms of design?

It is not only the broadcasters’ rethinking that is an issue; transfermedia has also been met with reservations about a possible sell-out of the film. A criticism that is understandable, says Wrogemann. In-stream shopping does not suit every format. It is not an option for arthouse films, but it is suitable for romantic comedies or series in which merchandising plays a role, for example. In addition, jay not only enables shopping, but also infotainment. With the rbb docu-drama “Kennedy’s Love for Europe”, jay has already realized a first use case. Thanks to the new content layer, viewers can access further information about the actors, historical events and locations directly in the player. This makes rbb the first German broadcaster to implement in-stream interaction in this way.

In addition to viewers as end customers, jay’s main target groups are distributors such as broadcasters and VoD platforms as well as producers. While jay’s technology opens up new levels of content for the audience, the B2B target group can generate sources of income via in-stream shopping and film productions also have the opportunity to finance themselves more broadly in advance. jay plans to implement the first major use case for in-stream shopping together with a broadcaster next year.

More about the MTH Blog

The media technologies of the future are already being used today – not only in the entertainment sector but in a wide variety of industries. For our MediaTech Hub Potsdam blog, we talk to tech enthusiasts, entrepreneurs and researchers once a month and tell the stories behind their innovative business models, ideas, projects and collaborations.