ideas4

romises and callbacks aren't easy to read or follow control flow * making a single threaded program parallel after writing it is almost a complete rewrite * work stealing thread pools solve the starvation problem, so they're good! * IO shouldn't block CPU and CPU shouldn't block IO * I am trying to design a syntax that is data orientated that means programs can be parallelised after writing them, so parallelisation comes for free * we can use the LMAX Disruptor pattern for efficient cross-thread communication. I use a lockfree multiconsumer multiproducer ringbuffer in my programs. I have an epoll-server which multiplexes clients/sockets over threads, this is more efficient than a thread-per-socket/client. I need to change it into a websocket server. Imagine you're a search engine company and you want to index links between URLs. How would you model this with async rust and thread pools? task download-url for url in urls: download(url) task extract-links parsed = parse(document) return parsed task fetch-links for link in document.query("a") return link task save-data db.save(url, link) How would you do control flow and scheduling and parallelism and async efficiently with this code? `db.save()`, `download()` are IO intensive whereas `document.query("a")` and `parse` is CPU intensive. I think its work diagram looks like this: https://github.com/samsquire/dream-programming-language/blob... I've tried to design a multithreaded architecture that is scalable which combines lightweight threads + thread pools for work + control threads for IO epoll or liburing loops: Here's the high level diagram: https://github.com/samsquire/ideas5/blob/main/NonblockingRun... The secret is modelling control flow as a data flow problem and having a simple but efficient scheduler. I wrote about schedulers here and binpacking work into time: https://github.com/samsquire/ideas4#196-binpacking-work-into... I also have a 1:M:N lightweight thread scheduler/multiplexer: https://github.com/samsquire/preemptible-thread

and queuing systems.

ringbuffer between pipes. It is inspired by prolog, whereby there can be multiple conditions or "facts" before a stateline "fires" and transitions. Transitions always go from left to right but within a stateline (what is between a pipe symbol) can fire in any order. A bit like a countdown latch. states = state1 | {state1a state1b state1c} {state2a state2b state2c} | state3 You can can think of each fact as an "await" but all at the same time. initial_state.await = { state1a.await state1b.await state1c.await }.await { state2a.await state2b.await state2c.await } | state3.await In io_uring and LMAX Disruptor, you split all IO into two halves: submit and handle. Here is a liburing state machine that can send and receive in parallel. accept | { submit_recv! | recv | submit_send } { submit_send! | send | submit_recv } I want there to be ring buffers between groups of states. So we have full duplex sending and receiving. Here is a state machine for async/await between threads: next_free_thread = 2 task(A) thread(1) assignment(A, 1) = running_on(A, 1) | paused(A, 1) running_on(A, 1) thread(1) assignment(A, 1) thread_free(next_free_thread) = fork(A, B) | send_task_to_thread(B, next_free_thread) | running_on(B, 2) paused(B, 1) running_on(A, 1) | { yield(B, returnvalue) | paused(B, 2) } { await(A, B, returnvalue) | paused(A, 1) } | send_returnvalue(B, A, returnvalue)

complete beginner to this space but I've been trying to understand the dining philosophers one. TLA+ Toolbox is aware of discrete states in the state space, which is absolutely awesome. Types can inform us about future possible valid states. I began writing a visualisation of memory and animated the movement of memory around to try reveal patterns. If you think of memory as state space, you can see movements of memory as picking things up and putting them down. https://replit.com/@Chronological/ProgrammingRTS#index.html If we see types or values as positions, we can create animations of the state space unfolding in front of us. This is the dream. (My plan is to write a programming simulation that is controlled similar to a real time strategy game as an alternative form of programming)

digms go (less powerful). But yes. > and more abstract No. Well, yes, if expressed with current programming languages. But that's part of the problem set, not part of the solution set. We should be able to express our architectures less abstractly, more concretely, but for that we need linguistic support. Which is why I am working on that: http://objective.st > I want software architecture to be cheap and easy to change without breaking any existing behaviours. I don't know much research on this subject. There was quite a bit of research at CMU, for example on packaging mismatch. Famous paper Architectural Mismatch, Why Reuse is so hard, and the 10 year follow up in 2009: Architectural Mismatch: Why Reuse is Still So Hard https://repository.upenn.edu/cgi/viewcontent.cgi?article=107...* Not much has changed since. > https://github.com/samsquire/ideas4 > https://devops-pipeline.com Will check those out. Dataflow is definitely a big part of it, with the extension of dataflow constraints (make, spreadsheets, "FRP"/"Rx"). But so is in-process REST with Storage Combinators! And breaking down barriers between scripting and "real" programming.

urpose transformations. If you have a codebase that has a client per thread and you want to refactor it to multiplex clients over threads. Or you want to add load shedding or backpressure to your microservice architecture, you have a large amount of work to do. If you want to add a field to a message or underlying library that your every microservice architecture uses, you have a lot of pain to go through. If you want to change the workflow of a digitised business procedure you have a LOT of refactoring to do. My ideas on the solution to this problem include: - Vector programming: represent software architecture as vector embeddings, code against sequences of vectors that are arranged into relationships and then the space where the vectors reside is concretised - Compositional behavioural typing: We need some way of combining behaviours together, composing the code of functions such that code is interleaved rather than chaining together functions. In most programming languages I know, data is typed, not the behaviour, and behaviour cannot be changed without changing the code. I think behaviour is more useful to type than data. Composing behaviour is not g(f(x)), it's taking arbitrary parts of the innerworkings of f and g and applying behaviours to their mechanisms. Inheritance doesn't do it properly, because you have to rewrite functions to replicate the original behaviour with a tweak. React hooks don't do it. It's the missing idea of computer science. My dream is that classes are compatible if you can provide a mapping from the innerworkings of one class to another class and the code can seamlessly compose without rewriting. And it be open to extension. It's like metaprogramming for behaviour rather than types. Code that that matches a "dispatcher" interface, automatically inferred by the compiler, and I can can insert a Load balancer or an ACL arbitrarily in the stack, anywhere. I can change the plurality (one-to-many, many-to-one) of architectural relationships, such as insert a hierarchy of load balancers here between these objects. I can make my software internet scalable, multiregion, multithreaded by adding to the diagram. I can see the architecture as a giant relational diagram data structure and mutate it directly and cheaply and without breaking any existing behaviour. I've been journalling about this on github.com/samsquire/ideas4 I also wrote https://devops-pipeline.com which is an dataflow/diagram based perspective of software infrastructure.

ng on designing interthread communication patterns for multithreaded software. How do you design an architecture that is easy to change, scales and is flexible? I am thinking of a message routing definition format that is extremely flexible and allows any topology to be created. https://github.com/samsquire/ideas4#526-multiplexing-setting... I think there is application of the same pattern to the network layer too. Each communication event has associated with it an environment of keyvalues that look similar to this: petsserver1 container1 thread3 socket5 user563 ingestionthread1 These can be used to route to keyspace ranges (such as particular users to tenant shards or load balance) to other components. For example users1-1000 are handled by petsserver1 and socket5 is associated with thread3. In other words: changing the RabbitMQ routing settings doesn't change the architecture of your software. You need to change the architecture of the software to match the routing configuration. But what if you changed the routing configuration and the application architecture changed to match?

ots of microtasks and switch between them concurrently with a switch statement. You switch between tasks with a scheduler loop, so you do a bit of work on each task independently. It's concurrent but not parallel. here's my writeup of concurrent loops which takes arbitrarily nested loops and turns them into an iterator https://github.com/samsquire/ideas4#133-concurrent-loops---l... here's my java state machine of multithreaded async/await in Java, that could be adapted to Javascript except it wouldn't be parallel. https://github.com/samsquire/multiversion-concurrency-contro...

vestigation. One of my journal entries is almost a lamentation of all the things that need to be done to release and use software. This is that entry: https://github.com/samsquire/ideas4#5-permanent-softwareplat... I wonder if software could be deployed more like a URL that has all the information to configure a virtual machine. Docker over URL or something.