Rendered at 13:38:04 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
aesthesia 1 days ago [-]
This is a neat little trick, but I wonder if you could do substantially the same thing by just prompting/LoRA finetuning the model to produce a single-token output ("yes" or "no"). This only requires a single model forward pass, you can use the same KV caching strategy for shared parts of the prompt, and isotonic regression should work just as well to calibrate the output logits. I guess if you use this method and probe on an internal layer you can skip all the remaining layers, which could be a nice inference speedup.
wren6991 22 hours ago [-]
> you could do substantially the same thing by just prompting/LoRA finetuning the model to produce a single-token output ("yes" or "no")
You could probably achieve this with logit masking. Or equivalently, comparing the "yes" vs "no" logprobs in the final dis-embedded vector.
You could probably achieve this with logit masking. Or equivalently, comparing the "yes" vs "no" logprobs in the final dis-embedded vector.