webcodecs

hi HN! I've been working in the browser audio domain recently and noticed to my surprise that WebAudio has no way to decode audio files efficiently. WebAudio has no support for partial decoding and can only decode entire audio files upfront which results in huge CPU and memory usage. For example, decoding a typical stereo 44.1 KHz 1-hour long audio file (e.g a podcast) into uncompressed floating point samples would consume (3600s * 44100Hz * 4 bytes * 2 channels) = ~1.27 GB of memory. Nice Obviously this isn't acceptable so I ended up creating this library to work around this issue. I ported FFmpeg over to WASM using Emscripten (a painful process) and then implemented an audio decoder with support for partial decoding. I hope this library can help anyone else with the same issue or at least serve as an additional reference for those look

Hey, I used to work on this :). It's complicated :). Basically, WebRTC is a combination of a bunch of protocols: ICE, DTLS, SCTP, and RTP. You could theoretically reduce that to ICE + QUIC for p2p use cases and just QUIC for client/server use cases. For p2p, there is RTCQuicTransport (see https://developers.google.com/web/updates/2019/01/rtcquictra...) For client/server, there is QuicTranport (see https://web.dev/quictransport/) Of course, you'll probably want to be able to encode and decode some audio and video as well to make that useful. For that, there is WebCodecs (see https://github.com/WICG/web-codecs or https://www.chromestatus.com/feature/5669293909868544) For use cases like live streaming and cloud gaming, I did a presentation about the combination of WebTransport + WebCodecs: https://www.w3.org/2018/12/games-workshop/slides/

It depends on what you mean by "WebRTC". There's the mobile/native library (at webrtc.org) and the standard (at https://www.w3.org/TR/webrtc/). The library does support it (although you have to write your own signaling and media forwarding server). The standard does not (yet). But it's being worked on ine the W3C (for example, Insertable Media Processing like this: https://www.w3.org/2019/09/18-mediaprocessing-harald-inserta... or something more low-level like this: https://github.com/WICG/web-codecs/blob/master/explainer.md). But the web standard not being there yet doesn't prevent someone from making a mobile or desktop app with e2ee conferencing, like Duo does: https://support.google.com/duo/answer/9280240?hl=en. (I used to work at Google on WebRTC, Duo, Hangouts, and WebCodecs, but now work on video calling at Signal).

Hi there, I'm from Chrome, someone on my team pointed me at this thread. My team supports casting features in Chrome. This is a fantastic project and it's great to see energy around this. I get the frustration around the inability to make devices work together for casting. In the standards community, we have been working with other companies and researchers on a protocol [1] that provides the groundwork for interoperability in media streaming and media control on the Web between browsers/apps and devices. We are developing an open source implementation and plan to see it in future products. There's a lot of other work going on in the Web standards community as well, that in the near future will allow Web apps to create their own media codec pipelines and streaming protocols customized to their needs [2] [3]. [1] https://w3c.github.io

There's a proposal that would let video encoding be accelerated without going through the entire WebRTC pipeline: https://github.com/WICG/web-codecs/blob/master/explainer.md Somewhat tangentially related, for cases when P2P connectivity is not needed, there's also a proposal that lets you send data unreliably, but in a client-server setting: https://github.com/WICG/web-transport/blob/master/explainer.... Both are currently being actively prototyped in Chromium (full disclosure: I've been working on the latter).