Wav2Lip

ables interesting features for many products – you can use it to seamlessly translate videos from one language to another, create personalized ads / video messages to send to your customers, or clone yourself so you never have to record a piece of content again. We’re excited about this area of research / the models we’re building because they can be impactful in many ways: [1] we can dissolve language as a barrier check out how we used it to dub the entire 2-hour Tucker Carlson interview with Putin speaking fluent English: https://vimeo.com/914605299 imagine millions gaining access to knowledge, entertainment, and connection — regardless of their native tongue. realtime at the edge takes us further — live multilingual broadcasts + video calls, even walking around Tokyo w/ a Vision Pro 2 speaking English while everyone else Japanese. [2] we can move the human-computer interface beyond text-based-chat keyboard / mice are lossy + low bandwidth. human communication is rich and goes beyond just the words we say. what if we could compute w/ a face-to-face interaction? Many people get carried away w/ the fact LLMs can generate, but forget they can also read. The same is true for these audio/visual models — generation unlocks a portion of the use-cases, but understanding humans from video unlocks huge potential. Embedding context around expressions + body language in inputs / outputs would help us interact w/ computers in a more human way. [3] and more powerful models small enough to run at the edge could unlock a lot: eg. extreme compression for face-to-face video streaming enhanced, spatial-aware transcription w/ lip-reading detecting deepfakes in the wild on-device real-time video translation etc. We’re building + scaling an API / SDK to bring this capability directly to the apps / services people already use. Lip-syncing is the first step, but we’re moving towards generating / modifying facial expressions, speech, and eventually a foundational approach to modify a human in video in any way you can imagine. Our playground and API is live today – we’re early, but we’re iterating quickly. We’d love any feedback you have on the playground experience, API dev/ex, and the quality of the output of our models :) we appreciate you and your time. You can play with it here for free: https://app.synclabs.so/