ChatRTX

The software ChatWithRTX is based on can be rebuild and might be able to be reconfigured to work: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/TensorRT-LLM

Having installed this, this is an incredibly then wrapper around the following github repos: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/TensorRT-LLM It's quite a thin wrapper around putting both projects into %LocalAppData%, along with a miniconda environment with the correct dependnancies installed. Also for some reason the LLaMA 13b (24.5GB) and Ministral 7b (13.6GB) but only installed Ministral? Ministral 7b runs about as accurate as I remeber, but responses are faster than I can read. This seems at the cost of context and variance/temperature - although it's a chat interface the implementation doesn't seem to take into account previous questions or answers. Asking it the same question also gives the same answer. The RAG (llamaindex) is okay, but a little suspect. The installation comes with a default folde

The source is available, minus the installer. You could always use the base repo after verifying it: https://github.com/NVIDIA/trt-llm-rag-windows