webcodecs

Ah, I didn't realize that always happened. I thought it was only if you did something that might have OS specific rendering characteristics (text-draws, etc). Maybe having an ImageEncoder API might be worthwhile after all then https://github.com/w3c/webcodecs/issues/204.

So basically webcodecs API is the best thing to happen in video processing, decoding and encoding are the most demanding parts of rendering, honestly there werent any good alternatives at all and If it wasnt for webcodecs api I wouldnt even bother trying to do it fully clientside, it would be slow and all, here is a bit about the alternatives: https://github.com/w3c/webcodecs/blob/main/explainer.md none of those alternatvies were specifically just for decoding and encoding, usually you had to work around it and get average results at best which I really wasnt satisfied with. When it comes to ffmpeg vs webcodecs API, honestly ffmpeg is slow as decoder and encoder I really dont recommend using it when theres webcodecs api, obviously depends what you are trying to do, but my god I dont remember exacly how fast it was but ive got like 10 fps

Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672 Thanks for the great work on WebCodecs!

* Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind? * Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here. * Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.

The amount of frames necessary is going to depend on the codec and bitstream parameters. If it's H264 or H265, there's some more discussion and links here: https://github.com/w3c/webcodecs/issues/698#issuecomment-161... The optimizeForLatency parameter may also help in some cases: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecode...

https://blog.paul.cx/post/audio-video-synchronization-with-t... has some background, https://github.com/w3c/webcodecs/blob/main/samples/lib/web_a... is part of a full example that you can run using web codecs, web audio, audioworklet SharedArrayBuffer, and does A/V sync. If it doesn't answer your question let me know because I wrote both (and part of the web audio spec, and part of the webcodecs spec).

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues

From developer.mozilla.org [0] > WebCodecs API > The WebCodecs API gives web developers low-level access to the individual frames of a video stream and chunks of audio. It is useful for web applications that require full control over the way media is processed. For example, video or audio editors, and video conferencing. And from w3c [1]: > The WebCodecs API allows web applications to encode and decode audio and video All this looks really promising, I wouldn't have thought that we could use browsers directly to render videos. Maybe Puppeteer could then stream the content of the page it is rendering, for example a three.js animation. [0] https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_A... [1] https://github.com/w3c/webcodecs

I work for PaperlessPost.com, and for our Flyer product we export mp4s from ffmpeg compiled to WASM, which is similar to ffmpeg.js but optimized for our use case which sounds similar. We have this WebAssembly method working well but iOS 15 (and the other major browsers) now support MediaExporter which might be a better way to go if you have something else to convert these files. This becomes and issue because you can't control the format MediaRecorder is recording to but the management of memory, the image quality, the compression, the performance hit will be more ideal than these other methods. OMGgif is very slow and will produce large files or very bad looking ones. Keep in mind that GIFs limit the colors so something that looks nice on the screen might not look the same after it is saved. The other thing to look out for is the WebCode

hi HN! I've been working in the browser audio domain recently and noticed to my surprise that WebAudio has no way to decode audio files efficiently. WebAudio has no support for partial decoding and can only decode entire audio files upfront which results in huge CPU and memory usage. For example, decoding a typical stereo 44.1 KHz 1-hour long audio file (e.g a podcast) into uncompressed floating point samples would consume (3600s * 44100Hz * 4 bytes * 2 channels) = ~1.27 GB of memory. Nice Obviously this isn't acceptable so I ended up creating this library to work around this issue. I ported FFmpeg over to WASM using Emscripten (a painful process) and then implemented an audio decoder with support for partial decoding. I hope this library can help anyone else with the same issue or at least serve as an additional reference for those look

Hey, I used to work on this :). It's complicated :). Basically, WebRTC is a combination of a bunch of protocols: ICE, DTLS, SCTP, and RTP. You could theoretically reduce that to ICE + QUIC for p2p use cases and just QUIC for client/server use cases. For p2p, there is RTCQuicTransport (see https://developers.google.com/web/updates/2019/01/rtcquictra...) For client/server, there is QuicTranport (see https://web.dev/quictransport/) Of course, you'll probably want to be able to encode and decode some audio and video as well to make that useful. For that, there is WebCodecs (see https://github.com/WICG/web-codecs or https://www.chromestatus.com/feature/5669293909868544) For use cases like live streaming and cloud gaming, I did a presentation about the combination of WebTransport + WebCodecs: https://www.w3.org/2018/12/games-workshop/slides/

It depends on what you mean by "WebRTC". There's the mobile/native library (at webrtc.org) and the standard (at https://www.w3.org/TR/webrtc/). The library does support it (although you have to write your own signaling and media forwarding server). The standard does not (yet). But it's being worked on ine the W3C (for example, Insertable Media Processing like this: https://www.w3.org/2019/09/18-mediaprocessing-harald-inserta... or something more low-level like this: https://github.com/WICG/web-codecs/blob/master/explainer.md). But the web standard not being there yet doesn't prevent someone from making a mobile or desktop app with e2ee conferencing, like Duo does: https://support.google.com/duo/answer/9280240?hl=en. (I used to work at Google on WebRTC, Duo, Hangouts, and WebCodecs, but now work on video calling at Signal).

Hi there, I'm from Chrome, someone on my team pointed me at this thread. My team supports casting features in Chrome. This is a fantastic project and it's great to see energy around this. I get the frustration around the inability to make devices work together for casting. In the standards community, we have been working with other companies and researchers on a protocol [1] that provides the groundwork for interoperability in media streaming and media control on the Web between browsers/apps and devices. We are developing an open source implementation and plan to see it in future products. There's a lot of other work going on in the Web standards community as well, that in the near future will allow Web apps to create their own media codec pipelines and streaming protocols customized to their needs [2] [3]. [1] https://w3c.github.io

There's a proposal that would let video encoding be accelerated without going through the entire WebRTC pipeline: https://github.com/WICG/web-codecs/blob/master/explainer.md Somewhat tangentially related, for cases when P2P connectivity is not needed, there's also a proposal that lets you send data unreliably, but in a client-server setting: https://github.com/WICG/web-transport/blob/master/explainer.... Both are currently being actively prototyped in Chromium (full disclosure: I've been working on the latter).