Posted by
A few days after Kuaishou Technology, a Chinese company known for its short-video platform, dropped its text-to-video model named Kling, there is a new contender on the block. US-based startup Luma AI which focusses on visual AI has introduced a new video generator similar to OpenAI’s Sora. Luma AI has introduced a new tool named Dream Machine.
Unlike Sora and Kling, Dream Machine offers access to the public. The company claims that Dream Machine is a next-generation video model that is capable of creating high-quality, realistic shots from prompts in natural language using AI. “Dream Machine is an AI model that makes high quality, realistic videos fast from text and images,” reads the introductory text on the official website.
Introducing Dream Machine – a next generation video model for creating high quality, realistic shots from text instructions and images using AI. It’s available to everyone today! Try for free here https://t.co/rBVWU50kTc #LumaDreamMachine pic.twitter.com/Ypmacd8E9z
— Luma AI (@LumaLabsAI) June 12, 2024
According to Luma AI, Dream Machine is a highly scalable and efficient transformer model trained directly on videos which makes it capable of generating physically accurate shots. The company claims that Dream Machine is its first step towards building a universal imagination engine. The tool is now available to everyone.
The company claims that Dream Machine generates 120 frames in 120s and it can iterate faster. The tool reportedly generates five second shots with smooth motion, cinematography, and drama. Besides the tool, it can understand how humans, animals, and objects interact with the physical world allowing it to create videos with great character consistency and accurate physics. The tool also helps in experimentation with an endless array of fluid, cinematic, and naturalistic camera motions.
Dream Machine is not without its limitations. As of now the official website lists morphing, movements, text, and Janus as its limitations.
While Sora generates high-definition videos lasting up to one minute using prompts in natural language, Kling is capable of creating videos that last up to two minutes. Since Dream Machine is free to use, we signed up and attempted to create a short video. We used the prompt, “Peter Pan flying on a carpet between galaxies.” It took a long time to create the video, about an hour. The output was bizarre, with Peter Pan wearing a dress and floating with his fingers warping, and there was no carpet either.