Cuebric Generative AI: Now We Are All Image Makers

In the future the director of a piece of entertainment might be able to shape their vision with holograms to create a realistic 3D simulation of anything they would like to see. Perhaps the ‘director’ is also the consumer, in the sense that content can be created, live, and experienced by an individual participant simply with a few voice prompts.

This where we are going with Generative AI, if you believe Pinar Seyhan Demirdag, Co-Founder (with Gary Lee Koepke) of Seyhan Lee, a Boston-based start-up which claims to have developed the first virtual production tool that runs on Gen-AI.

“I would like to imagine a world where no matter your background or talent, anyone can play a central role in the filmmaking process,” she told IBC365. “A world where their vision is facilitated and co-produced by AI tools in real time.”

Seyhan Lee’s Cuebric enables filmmakers to dream up camera-ready media in seconds, for instance to playback in an LED volume. Fifteen studios are experimenting with the technology and it was on track to return the founder’s initial investment earlier this year, in double-quick time.

The business case for Cuebric

It is one of many AI tools being introduced to the industry as a means of cutting costs and, the developers argue, freeing up time for creatives to be creative.

“Let’s start with notion of waiting on set,” Demirdag said. “We always feel we’re rushing so we can collectively wait. With a realtime environment generation tool like Cuebric the director can change their mind in realtime and make, I would argue, the world’s most collaborative artform even more collaborative. It energises people to be more creative on set.

Read more Interview: Douglas Rushkoff on AI in Media

A second business case for Cuebric is that it consigns manual rotoscoping (the process of tracing over live-action footage frame by frame) to history. “Rotoscoping is one of the most tedious and time-consuming parts of the filmmaking process. If it were to disappear from the face of the planet no one would argue.”

Currently, Cuebric enables 2D rotoscoping but future upgrades would, for example, allow users to select a 3D character and remove her from every frame in a scene.

Putting exact figures on the costs saved when using a new tool like this is tricky but Seyhan Lee have identified one area where producers can realise instant financial benefits.

“Five percent to 20% of budgets today goes into reshoots because even when you’re working with green screen or virtual production, reshoots are often the only way to achieve the director’s intent.”

Seyhan Lee calculate that for a medium size picture, reshoots cost a production $375,000 on average. For bigger budget shows that rises to an astonishing $24 million.

“If we were to save the industry even a fraction of that using AI it could help funnel those funds back into creativity and cut out unnecessary labour,” Demirdag said.

Cuebric is a website [cubebric.com] allowing uses to write prompts to generate synthetic media. All the data processing requires banks of GPUs in order to render photoreal imagery at speed and runs in the cloud. It bundles together five algorithms. One of them will turn existing photographic plates into camera ready LED screen definitions, up to 16K. You can add depth to scenes and also choose between Classic, Moody and Sci-Fi modes for cinematic looks. It’s main ‘trick’, though, is to output original pixel-ready environments, VFX or concept art trained on Stable Diffusion dataset LAION.

All of LAION’s image datasets are built off of Common Crawl, a nonprofit that scrapes billions of webpages monthly and releases them as massive datasets.

“This is a dataset trained on the total sum of humanity, any picture that ever appeared online and not the artwork of one person,” Demirdag explained.

In the US, developers of leading Gen-AI tools including Stable Diffusion and ChatGPT owner OpenAI are battling lawsuits from stock photo agencies and groups of artists alleging that their copyright has been infringed.

Image training data and copyright

The US Supreme Court recently found against the estate of pop artist Andy Warhol in favour of the photographer on which he based a piece of commissioned artwork. The ruling is considered to significant for how future litigation will be decided in favour of artists and against the ‘fair-use’ argument made by AI tools developers.

With no universal convention on AI copyright, the Japanese government recently decided to adopt the opposite approach. It will not enforce copyright on data used in AI training stating that to do otherwise would be to hold back the nation’s progress in AI technology.

For Demirdag that is very important, she said. “What the [Japanese] signal is that Gen-AI is a natural progress for human development and by pausing it with copyright law they will be stopping human progress.”

That said, Demirdag is no advocate for plagiarism. If Cuebric were to be trained to generate Warhol-style artworks without seeking his estate’s permission “it would simply be wrong,” she declared. “It is not the style of entrepreneurship that I choose.”

There is a very big difference, she insisted, in training a data set on images “from the total sum of humanity” versus a data set based on the works of a particular artist or group of artists.

“Currently there is a chaos and confusion in the public’s mind in being able to understand the difference between the two.”

AI is just a tool

There is also paranoia at the pace of AI’s advance into the industry. The potential of Generative AI to automate almost every part of the filmmaking process has become an existential crisis for some in Hollywood and the backdrop to the writer’s strike earlier this year.

Demirdag is adamant that AI is a tool that no-one should be fearful of if they take a step back and view the technology objectively.

“AI on its own is nothing,” she said. “The more we personalise AI the more it subconsciously dehumanises humans. It is the responsibility of every single human being to research what AI does do. It’s actually quite simple. There’s a data set and there’s an algorithm and it produces results in order to serve your creativity. The AI does not create. We create by using AI.”

As an example, she points to the bullet time sequence developed painstakingly from research into photogrammetry in 1998 and employed in The Matrix a few years later. “Now it’s something anyone can basically do now on a smartphone,” she said.

She believes Generative AI will turn the creation of motion pictures into a realtime process and enable an auteur’s vision to be realised much more accurately.

“Right now, filmmakers conceptualise, pre-produce, produce and post-produce in a linear fashion with many weeks of lost time in between. Imagine a production process where all that collapses into one, where everyone collaboratively exchanges ideas in the now. Imagine testing ideas and seeing a vision come to life in realtime.”

The future of image generation

Dermirdag, who is Turkish, has had a fascinating route to media and entertainment, the through-line being graphics, art and design. She’s designed patterns for IKEA (and has a 25-piece collection of furniture under the name Pinar&Viola), worked on campaigns for AirBnB, made a stained glass window for a Jewish synagogue and designed a holographic catwalk for a virtual fashion line. Several of her paintings are hung in galleries and she claims to be one of first anywhere to use face tracking technology (in 2011, two years before Snapchat).

“I am image generator on my own,” she said. “I am trained to generate different images for different surfaces. I don’t know how to read or write code but I have an intuitive knowledge about where image generation is going.”

She does not envisage that key crafts of the cinematographer, actor, director or editor will disappear. Instead, existing technologies like the camera will remain in play just as the paintbrush and canvas have lasted a thousand years.

“The problem is not that artists will be obsolete. Quite the reverse. Crafts people will rise to the surface and their work will still be valued. Everybody maybe qualified to make a film but this does not mean that everything they produce will be great.”

She warned of the “danger of normalisation of mediocrity” where AI churns out formulaic content but said we’ve nothing to fear because art and artists will rise to the surface and still be valued.

“Almost everyone has a digital camera in their pocket but does it mean that the vast majority of photos they produce are any good? Only a few photographers have the talent and skill to use the technology to produce work of value.”

Demirdag is in no doubt: “Generative AI has nothing to do with creativity. It has everything to do with being your parallel processing, never tiring, just your assistant constantly giving you options for you to curate, review, and select.”

Read more AI in Broadcast