see_proto

Links: - Repo: https://github.com/kodomonocch1/see_proto - Release (Demo + DD packs): https://github.com/kodomonocch1/see_proto/releases/tag/v0.1.... - NDA / enterprise inquiry form: https://docs.google.com/forms/d/e/1FAIpQLScV2Ti592K3Za2r_WLU... If you want to verify quickly: 1) Demo: run the offline Demo Pack (~10 minutes) 2) DD: confirm pack_verify PASS + mismatch=0 + audit PASS (see the release notes checklist)

Hi HN — I built SEE (Semantic Entropy Encoding), a schema-aware, searchable compression format for JSON/NDJSON. Goal: Reduce the “data tax” (storage/egress) and the “CPU tax” (decompress/parse) by keeping data searchable while compressed. What’s different: - Page-based layout + Bloom filter skipping + a small directory index - Fast exists/pos/eq-style lookups without full decompression Proof-first (no meeting required): - A 10-minute offline Demo Pack (wheel + sample .see + scripts + OnePager) - A DD Evidence Pack designed for reviewers (mismatch-zero checks, audit status, verification file list) I’m exploring either: (A) acquisition (asset purchase), or (B) an exclusive license with a strategic buyer with a clear integration path. Evaluation slots are limited. Repo + release assets are linked in the comments. Happy to answer technical qu

I built SEE (Semantic Entropy Encoding): a page-level JSON/NDJSON format that stays searchable while compressed (exists/pos/eq-style probes), using Bloom+skip + structure-aware encoding. On our GitHub events dataset, SEE ended up smaller than Zstd-19 while still supporting random access queries: - combined: 40.4MB vs Zstd 71.8MB (raw 524.1MB) → 7.7% of raw - str: 9.1MB vs Zstd 9.5MB - int: 31.3MB vs Zstd 62.3MB Lookup microbench (one column): p50 ~0.085ms. Repo + release assets are here: https://github.com/kodomonocch1/see_proto NDA eval request (optional): https://docs.google.com/forms/d/e/1FAIpQLScV2Ti592K3Za2r_WLU... Happy to answer questions about the design trade-offs and where this beats “Zstd + separate index”.

Problem JSON/NDJSON is everywhere, but once you compress, you usually lose searchability. You either keep raw/queryable stores (high I/O and egress) or compress to gz/zstd blobs (cheap to store, painful to probe). That “cloud tax” shows up as wasted reads and time. What I built (SEE — Semantic Entropy Encoding) A schema-aware, searchable compression codec that keeps `exists` / `pos` lookups in sub-ms while still compressing. Internals: structure-aware delta + dictionaries, a PageDir + mini-index to jump to relevant pages, and a tuned Bloom filter that skips ~99% of pages. AutoPage (131/262 KiB) balances seek vs throughput. Benchmarks (apples-to-apples on a desktop: i7-13700K/96GB/Windows) • Size ratio: strings ≈ 0.168–0.170; combined ≈ 0.194–0.196 • Bloom density ≈ 0.30; skip: present ≈ 0.99, absent ≈ 0.992 • Lookup latency (ms): pres

Problem JSON/NDJSON is everywhere in data platforms, but compression usually breaks searchability. You either keep queryable raw stores (high I/O/egress) or compress into gz/zstd blobs (cheap to store, painful to probe). The “cloud tax” shows up as wasted reads. What I built (SEE — Semantic Entropy Encoding) A schema-aware, searchable compression codec for JSON that keeps exists/pos lookups fast while still compressing. Internals: structure-aware delta + dictionaries, a PageDir + mini-index to jump to relevant pages, and a tuned Bloom filter that skips ~99% of pages. AutoPage (131/262 KiB) balances seek vs throughput. Benchmarks (apples-to-apples, FULL) - size ratio: str ≈ 0.168–0.170, combined ≈ 0.194–0.196 - Bloom density ≈ 0.30; skip: present ≈ 0.99, absent ≈ 0.992 - lookup (ms): present p50/p95/p99 ≈ 0.18/0.28/0.37; absent ≈ 1.16–1.88

SEE (Semantic Entropy Encoding) is a schema-aware JSON compression format. It keeps JSON searchable while compressed, cutting I/O and CPU cost. Benchmarks: ~19.5% of raw size, lookup p50 ≈ 0.18 ms. Article: https://medium.com/@tetsutetsu11/the-hidden-cloud-tax-and-th... Slides: https://speakerdeck.com/tetsu05/see-the-hidden-cloud-tax-bre... GitHub: https://github.com/kodomonocch1/see_proto Curious about your thoughts — especially from those using Zstd or Parquet in production. What would be your biggest blocker for adopting schema-aware compression?

Hi! I built SEE (Semantic Entropy Encoding) because the “data tax” (storage/egress) and the “CPU tax” (decompress/parse) keep rising together. Tradeoff: it’s not always smaller than Zstd, but it stays searchable while compressed and minimizes I/O. Key numbers (demo): combined≈19.5% of raw, skip≈99%, lookup p50≈0.18 ms (bloom≈0.30). 10-min reproduction (no marketing): 1) Download the Demo ZIP (Release). 2) Follow README_FIRST.md. 3) Run `python samples/quick_demo.py` → prints ratio/skip/bloom + p50/p95/p99. ROI quick math: Savings/TB ≈ (1 − 0.195) × Price_per_GB × 1000 (e.g., $0.05/GB → ~$40/TB). NDA/VDR (private, no confidential info in public): [https://docs.google.com/forms/d/e/1FAIpQLScV2Ti592K3Za2r_WLU...] Happy to answer technical questions (schema-aware layout, delta strategy, bloom density, skip heuristics, failure modes).