conc

waitgroup's inadequacy is why I end up using structured concurrency like https://github.com/sourcegraph/conc

> Actually much closer than anything I saw in other mainstream languages eg Java or Go. https://github.com/sourcegraph/conc

Synchronization primitives in Go are just as divorced as elsewhere, sometimes even more so - it does have channels, but Goroutines cannot yield a value, forcing you to employ a separate storage location together with WaitGroup/Mutex/RWMutex (which, unlike Rust's RWLock, is separate too, although C# lets you model it to an extent). This results in community developing libraries like https://github.com/sourcegraph/conc which attempt to replicate Rust's Futures / C#'s Tasks.

Looks good, similar to https://github.com/sourcegraph/conc which we've been using for a while. Will give this a look.

Sourcegraph Conc is broadly similar in providing pool helpers, but doesn't provide the same fine grained batching options: https://github.com/sourcegraph/conc Uber CFF does code generation, and has more of a focus on readability and complex dependency chains: https://github.com/uber-go/cff

JS concurrency is crap. It should be shot and buried in a lead coffin. Debugging async code is pure hell. With Go, you have a normal debugger that can be used to step over the code. You can get normal stack traces for all threads if needed. There is a race detector that can catch most of unsynchronized object access. With JS? You're on your fucking own. You can't find out the overall state of the system ("the list of all threads"), without getting deep into the guts of React or whatever framework you're using. Debugger is useless, as each `await` call drops you into the event loop. So pretty much every complicated non-trivial JS app ends up with _tons_ of race conditions, by depending on the order of async functions finishing. Speaking of race conditions. Coming from classic multithreading and Go, I tried to use `Promise.race` to simulate

Not speaking on whether the language should make this easier without an external library, but wouldn't https://github.com/sourcegraph/conc help in that scenario? It has context-aware and error-aware goroutine pools, seems like the exact fit for what you are trying to do. Although admittedly I dive too deep into your code.

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g... There's a library for it: https://github.com/sourcegraph/conc But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new language can be valuable, even if it has no "new" language features, just to get a chance to reboot the standard library it has and push for patterns that older languages are theoretically capable of, but they just don't play well with any of the libraries in the language. Having it as a much-later 3rd party library just isn't good enough. (In fact if I ever saw a new language start up and that was basically its pitch, I'd be v

What are the possible use cases for this tool? Why would I want to find out the total runtime of all videos in a directory? Also, you might wanna limit concurrency[0] instead of spawning many ffprobe instances at the ~same time. [0]: https://github.com/sourcegraph/conc In another note, ChatGPT suggests this shell command to do the same thing. It doesn't process files in parallel though. find . -name "*.mp4" -print0 | \ xargs -0 -I {} ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 {} | \ awk '{sum += $1} END {print sum}'

While virtual threads will be stable in Java 21, Structured Concurrency is still a preview feature. You probably won't see it in production anytime soon. Preview features require a special flag when compiling and running them, and they won't run on newer versions of the JVM. I don't expect to see StructuredTaskScope in common production use before the next LTS version is out. But it doesn't mean you cannot have structured concurrency before that. Even in language that mostly enforce Structured Concurrency like Kotlin, it's still a library feature. Even the original blog post which formulated this concept, described a library that implemented structured concurrency for Python[1]. You can pretty easily implement structured concurrency yourself by creating your own implementation of StructuredTaskScope, if you need it right now. You can even

https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L... // Map applies f to each element of input, returning the mapped result. func Map[T, R any](input []T, f func(*T) R) []R { res := make([]R, len(input)) ForEachIdx(input, func(i int, t *T) { res[i] = f(t) }) return res } Seems a little silly to farm that off to a custom func when you could just write the for-loop, but it's probably fine / may be no different in practice. Let's go see what ForEach does, to make sure: https://github.com/sourcegraph/conc/blob/main/iter/iter.go#L... // ForEachIdx is the same as ForEach except it also provides the // index of the element to the callback. func ForEachIdx[T any](input []T, f func(int, *T)) { numTasks := runtime.GOMAXPROCS(0) numInput := len(input) if numTasks > numInput