VAC-Memory-System

/wiki/5-months-at-war-with-Claude-CLI:-how-I-built-a-memory-system-(and-what-it-cost-me)

uns in < 3s per query on a single RTX 4090. LoCoMo is currently the hardest public long-term memory benchmark (5.880 real human–agent conversations, multi-hop, temporal, negation, etc.). Beating Mem0 official baseline by ~12–14 pp with fully open components feels pretty good. Would love feedback, especially from people who are also grinding on agent memory systems. My background: My path didn't start in an IT office, but in Columbus, Ohio, where I worked as a handyman after leaving my job on the cell towers. The decision came from necessity: I bought a powerful PC on installments and resolved to create something that would change my life. I had no experience, but I had an idea. Using Claude CLI as my sole mentor, I focused on architecture, not syntax. Over 4.5 months of work, I engineered and created the VAC Memory System. To prove its value, I tested it on the toughest RAG benchmark—LoCoMo. Today, my system shows an overall result of 80.1% and a phenomenal 87.78% in the "Commonsense" category. This is more than just code; it is the result of faith in an idea. I showed that by using modern tools, it is possible to achieve SOTA-level performance and create serious technology, regardless of your starting point. I highly anticipate your feedback.

as a first-pass filter to catch exact-match questions. Threshold: coverage ≥ 0.1 → keep top-30 4. Union strategy Instead of aggressively reducing the union, the system feeds 112–135 documents directly to a re-ranker. In practice this improved stability and prevented loss of rare but crucial documents. 5. Cross-Encoder reranking bge-reranker-v2-m3 Processes the full union (rare for RAG pipelines, but worked best here) Produces a final top-k used for answer generation 6. Answer generation GPT-4o-mini, used only for the final synthesis step No agent chain, no tool calls, no memory-dependent LLM logic Performance <3 seconds per query on a single RTX 4090 Deterministic output between runs Reproducible test harness (10×10 protocol) Why this worked Three things seemed to matter most: MCA-first filter to stabilize early recall Not discarding the union before re-ranking Proper dense embedding instruction, which massively affects BGE performance Notes LoCoMo remains one of the hardest public memory benchmarks: 5,880 multi-hop, temporal, negation-rich QA pairs derived from human–agent conversations. Would be interested to compare with others working on long-term retrieval, especially multi-stage ranking or cross-encoder heavy pipelines. Github: https://github.com/vac-architector/VAC-Memory-System

as a first-pass filter to catch exact-match questions. Threshold: coverage ≥ 0.1 → keep top-30 4. Union strategy Instead of aggressively reducing the union, the system feeds 112–135 documents directly to a re-ranker. In practice this improved stability and prevented loss of rare but crucial documents. 5. Cross-Encoder reranking bge-reranker-v2-m3 Processes the full union (rare for RAG pipelines, but worked best here) Produces a final top-k used for answer generation 6. Answer generation GPT-4o-mini, used only for the final synthesis step No agent chain, no tool calls, no memory-dependent LLM logic Performance <3 seconds per query on a single RTX 4090 Deterministic output between runs Reproducible test harness (10×10 protocol) Why this worked Three things seemed to matter most: MCA-first filter to stabilize early recall Not discarding the union before re-ranking Proper dense embedding instruction, which massively affects BGE performance Notes LoCoMo remains one of the hardest public memory benchmarks: 5,880 multi-hop, temporal, negation-rich QA pairs derived from human–agent conversations. Would be interested to compare with others working on long-term retrieval, especially multi-stage ranking or cross-encoder heavy pipelines. Github: https://github.com/vac-architector/VAC-Memory-System

uns in < 3s per query on a single RTX 4090. LoCoMo is currently the hardest public long-term memory benchmark (5.880 real human–agent conversations, multi-hop, temporal, negation, etc.). Beating Mem0 official baseline by ~12–14 pp with fully open components feels pretty good. Would love feedback, especially from people who are also grinding on agent memory systems. My background: My path didn't start in an IT office, but in Columbus, Ohio, where I worked as a handyman after leaving my job on the cell towers. The decision came from necessity: I bought a powerful PC on installments and resolved to create something that would change my life. I had no experience, but I had an idea. Using Claude CLI as my sole mentor, I focused on architecture, not syntax. Over 4.5 months of work, I engineered and created the VAC Memory System. To prove its value, I tested it on the toughest RAG benchmark—LoCoMo. Today, my system shows an overall result of 80.1% and a phenomenal 87.78% in the "Commonsense" category. This is more than just code; it is the result of faith in an idea. I showed that by using modern tools, it is possible to achieve SOTA-level performance and create serious technology, regardless of your starting point. I highly anticipate your feedback.

+ full reproducible benchmarking): https://github.com/vac-architector/VAC-Memory-System Happy to answer technical questions, discuss the architecture, or hear critiques.

accuracy, not 80.1%. The 16% failures are real limitations. Call to Action: Next Benchmarks I need your recommendations: I am looking for the toughest long-term conversation benchmarks you know. What else should I test the VAC Memory system on to truly prove its generalizability? GitHub: https://github.com/vac-architector/VAC-Memory-System I appreciate the honesty of the HN community and your help in validating my work.