Rendered at 01:06:49 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
edude03 3 hours ago [-]
Devin was the first ai developer harness I ever heard of — and in fact the first one I ever used as I had gotten an invite while it was in private beta. Now it seems like the product has faded (into irrelevance?) between other harnesses like cursor especially and all the major harnesses having a cloud mode. Is there some other area Devin has become really popular / has stayed competitive?
torginus 1 hours ago [-]
I was just thinking recently, isn't this fascination with harnesses kind of backwards?
I mean ChatGPT is the harness most of the time that's calling the tools, MCPs, etc. Putting a loop around it doesn't really seem like it would be something that gets you a lot of extra milage.
drooby 31 minutes ago [-]
It gets an incredible amount of mileage. And I believe it is the interface that allows the model to learn and eventually bake that intelligence into the model.
Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.
The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.
Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.
A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.
I mean ChatGPT is the harness most of the time that's calling the tools, MCPs, etc. Putting a loop around it doesn't really seem like it would be something that gets you a lot of extra milage.
Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.
The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.
Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.
A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.