llm-cluster

I'm trying to understand the clustering code but not doing too well. https://github.com/simonw/llm-cluster/blob/main/llm_cluster.... So does this take each row from the DB, convert to a numpy array (?), then uses an existing model called MiniBatchKMeans (?) to go over that array and generate a bunch of labels. Then add it to a dictionary and print to console.

I only got the clustering working this morning, so aside from playing around with it a bit I've not had any results that have convinced me it's a tool I should throw at lots of different problems. I mainly like it as another example of the kind of things you can use embeddings for. My implementation is very naive - it's just this: sklearn.cluster.MiniBatchKMeans(n_clusters=n, n_init="auto") I imagine there are all kinds of improvements that could be made to this kind of thing. I'd love to understand if there's a good way to automatically pick an interesting number of clusters, as opposed to picking a number at the start. https://github.com/simonw/llm-cluster/blob/main/llm_cluster....

There's a lot of stuff in this release. Don't miss the new llm-cluster plugin, which can both calculate clusters from embeddings and use another LLM call to generate a name for each cluster: https://github.com/simonw/llm-cluster Example usage: Fetch all issues, embed them and store the embeddings and content in SQLite: paginate-json 'https://api.github.com/repos/simonw/llm/issues?state=all&filter=all' \ | jq '[.[] | {id: .id, title: .title}]' \ | llm embed-multi llm-issues - \ --database issues.db \ --model sentence-transformers/all-MiniLM-L6-v2 \ --store Group those in 10 clusters and generate a summary for each one using a call to GPT-4: llm cluster llm-issues --database issues.db 10 --summary --model gpt-4