slopo

"Non-exact" may be interpreted in different ways, so let's look at the example report: https://github.com/rafal-qa/slopo/tree/main/doc/example-repo... * cluster-01.md has the highest similarity, and jscpd probably will detect this too. * cluster-10.md has the lowest similarity still above the threshold, and I don't think jscpd will detect this as clones. Because they are not clones, this is a false positive that needs to be discarded. But in other cases this kind of similar code may be worth acting on. I didn't compare with jscpd because I don't consider it a competition. Embedding-based duplication detection works differently, gives different results and has its own trade-offs.

Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents. Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper. "brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks. From my perspective, this

It looks like it works only on function bodies[1]. I'm not sure I understand why you would want to look at invoked callables code, though. Calling the same set of helper functions is already flagged; repeated code in helpers is flagged as well when those helpers are analyzed. Do you have a specific example where you'd like a function flagged as a duplicate based on the code it calls out to? [1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...

Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed. I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for co