LlamaGym

The easiest thing would be to use supervised finetuning on the LLMs you're trying to control (most open-source LLMs have some sort of off-the-shelf system for finetuning), combined with this system to control the output. I suppose there's nothing stopping you from writing an RL training system to alter the model weights other than needing to write a bunch of code, though... Maybe LlamaGym (https://github.com/KhoomeiK/LlamaGym/tree/main) could reduce the amount of code you need?