TensorRT-LLM

r dataset, containing text files of nvidia marketing materials. When I tried asking questions about the files, it often cites the wrong file even if it gave the right answer.

, etc you'll see they have many more optimizations in the works. In typical Nvidia fashion TensorRT-LLM runs on any Nvidia GPU (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM. It even works on their Jetson line of hardware. You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic. Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back and then some as a dividend via the software especially when anything close (assuming this even is) is another paid product/SaaS/whatever their monetization strategy is. At the risk of this turning into an Nvidia sales pitch Triton will do the same thing for absolutely any model via the ONNX, TensorRT, Pytorch, Tensorflow, OpenVINO, etc backends. I have an implementation generating embeddings via bge-large-v1.5 that's also the fastest thing out there. Same for Whisper, vision models, whatever you want. I feel like MK1 must be aware of TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup. [0] - https://github.com/NVIDIA/TensorRT-LLM [1] - https://github.com/triton-inference-server/tensorrtllm_backe... [2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read... [3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...