semantic_uncertainty

the method described by this paper does not > draw[ing] several samples and evaluating the entropy and/or disagreement between those draws the method from the paper (as I understand it): - samples multiple answers, (e.g. "music:0.8, musician:0.9, concert:0.7, actress:0.5, superbowl:0.6") - groups them by semantic similarity and gives them an id ([music, musician, concert] -> MUSIC, [actress] -> ACTING, [superbowl] -> SPORTS), note that they just use an integer or something for the id - sums the probability of those grouped answers and normalizes: (MUSIC:2.4, ACTING:0.5, SPORTS:0.6 -> MUSIC:0.686, SPORTS:0.171, ACTING:0.143) They also go to pains in the paper to clearly define what they are trying to prevent, which is confabulations. > We focus on a subset of hallucinations which we call ‘confabulations’ for which LLMs fluently make claims

Catching "hallucinations" is quite useful for many applications. I'm doing some research in mitigating effects of factual errors in LLM generated answers for public agencies, where giving a factually wrong answer may be illegal. If those could be detected (with sufficient accuracy), the system could simply decline to give an answer and ask the user to contact the agency. Training the models not to give wrong answers (or giving them less) in the first place would of course be even better. Unnecessary complications come also from the use of pre-trained commercial black-box LLMs through APIs, which is (sadly) the way LLMs are used in applications in vast majority of times. These could perhaps be fine tuned through the APIs too, but it tends to be rather fiddly and limited and very expensive to do for large synthetic datasets like would be us