smartfunc doesn't really treat the LLM as a compiler - it's not generating Python code to fill out the function, it's converting that function into one that calls the LLM every time you call the function passing in its docstring as a prompt.
A version that DID work like a compiler would be super interesting - it could replace the function body with generated Python code on your first call and then reuse that in the future, maybe even caching state on disk rather than in-memory.
I would like to try something like this in Rust:
- you use a macro to stub out the body of functions (so you just write the signature)
- the build step fills in the code and caches it
- on failures the, the build step is allowed to change the function bodies generated by LLMs until it satisfies the test / compile steps
- you can then convert the satisfying LLM-generated function bodies into a hard code (or leave it within the domain of "changeable by the llm")
It sandboxes what the LLM can actually alter, and makes the generation happen in an environment where you can check right away if it was done correctly. Being Rust, you get a lot of more verifications. And, crucially, keeps you in the driver's seat.
Yeah, I do think that LLMs acting as compilers for super high-level specs (the new "code") is a much better approach than chatting with a bot to try to get the right code written. LLM-derived code should not be "peer" to human-written code IMO; it should exist at some subordinate level.
The fact that they're non-deterministic makes it a bit different from a traditional compiler but as you say, caching a "known good" artifact could work.
I use something similar to this decorator (more or less a thin wrapper around instructor) and have looked a little bit at the codegen + cache route. It gets more interesting with the addition of tool calls, but I've found JSON outputs create quality degradation and reliability issues. My next experiment on that thread is to either use guidance (https://github.com/guidance-ai/guidance) or reimplement some of their heuristics to try to get tool calling without 100% reliance on JSON.
Treating it as a compiler is obviously the way right? Setting aside overhead if you’re using local models.. Either the code gen is not deterministic in which case you risk random breakage or it is deterministic and you decided to delete it anyway and punt on ever changing / optimizing it except for in natural language? Why would anyone prefer either case? Code folding works fine if you just don’t want to look at it ever.
I can see this eventually going in the direction of "bidirectional synchronization" of NL representation and code representation (similar to how jupytext allows you work with notebooks in browser or markdown in editor). But a single representation that's completely NL and deliberately throwing away a code representation sounds like it would be the opposite of productivity..
Very cool. I implemented something similar for personal use before.
At that time, LLMs weren't as proficient in coding as they are today. Nowadays, the decorator approach might even go further and not just wrap LLM calls but also write Python code based on the description in the Docstring.
This would incentivize writing unambiguous DocStrings, and guarantee (if the LLMs don't hallucinate) consistency between code and documentation.
It would bring us closer to the world that Jensen Huang described, i.e., natural language becoming a programming language.
People have been talking about natural language becoming a programming language for way longer than even Jensen Huang has been talking about it. Once upon a time, they tried to adapt natural language into a programming language, and they came up with this thing called COBOL. Same idea: "then the managers can code, and we won't need to hire so many expensive devs!"
And now the COBOL devs are retiring after a whole career . . .
But isn't it actually more like, COBOL lets you talk in COBOL-ese (which is kinda stilted), whereas LLMs let you talk in LLM-ese (which gets a lot closer to actual language)? And then since the skill cap on language is basically infinite, that this becomes a question of how good you are at saying what you want - to the extent it intersects with what the LLM can do.
COBOL was the best attempt that they could get to in the 1960s. It's the entire reason COBOL has things like paragraphs, things end with periods, etc. They wanted as much of an "English-like syntax" as possible.
The reason it looks so odd today is that so much of modern software is instead the intellectual heir of C.
And yeah, the "skill cap" of describing things is theoretically infinite. My point was this has been tried before and we don't yet know how the actual limitations of an LLM come close to that ideal. People have been trying for decades to describe things in English that still ultimately need to be described in code for them to work; that's why the software industry exists in the first place.
Many libraries with the same approach suffer the same flaw: can't easily use the same function with different LLMs at runtime (ie. after importing the module where it is defined).
I initially used the same approach in my library, but changed it to explicitly pass the llm object around and in actual production code it's easier/more flexible to use.
And it uses a Python concrete syntax tree (with https://pypi.org/project/libcst/) to apply changes to just the docstrings without risk of editing any other code.
Imagine this: It would be cool when these functions essentially boiled down to a distilled tiny model just for that functionality instead of an api call to foundation one.
Funny. I frequently give the LLM the function and ask it to make the doc string.
TBH I find doc strings very tedious to write. I can see how this would be a great specification for an LLM but I dont know that its actually better than a plain text description of the function since LLMs can handle those just fine and they are easier to write.
LLM uses plugins because then there's no need to sync new providers with the core tool. When a new Gemini feature comes out I ship a new release of https://github.com/simonw/llm-gemini - no need for a release of core.
I can wake up one morning and LLM grew support for a bunch of new models overnight because someone else released a plugin.
I'm not saying "LLM is better than LiteLLM" here - LiteLLM is a great library with a whole lot more contributors than LLM, and it's also been fully focused on being a great Python library - LLM has also had more effort invested in the CLI aspect than the Python library aspect so far.
I am confident that a plugin system is a better way to solve this problem generally though.
This is the way LLM-enhanced coding should (and I believe will) go.
Treating the LLM like a compiler is a much more scalable, extensible and composable mental model than treating it like a junior dev.
smartfunc doesn't really treat the LLM as a compiler - it's not generating Python code to fill out the function, it's converting that function into one that calls the LLM every time you call the function passing in its docstring as a prompt.
A version that DID work like a compiler would be super interesting - it could replace the function body with generated Python code on your first call and then reuse that in the future, maybe even caching state on disk rather than in-memory.
Yes, that would be indeed very interesting.
I would like to try something like this in Rust: - you use a macro to stub out the body of functions (so you just write the signature) - the build step fills in the code and caches it - on failures the, the build step is allowed to change the function bodies generated by LLMs until it satisfies the test / compile steps - you can then convert the satisfying LLM-generated function bodies into a hard code (or leave it within the domain of "changeable by the llm")
It sandboxes what the LLM can actually alter, and makes the generation happen in an environment where you can check right away if it was done correctly. Being Rust, you get a lot of more verifications. And, crucially, keeps you in the driver's seat.
Ah, cool, didn't read close enough.
Yeah, I do think that LLMs acting as compilers for super high-level specs (the new "code") is a much better approach than chatting with a bot to try to get the right code written. LLM-derived code should not be "peer" to human-written code IMO; it should exist at some subordinate level.
The fact that they're non-deterministic makes it a bit different from a traditional compiler but as you say, caching a "known good" artifact could work.
https://github.com/eeue56/neuro-lingo
You can even pin the last result:
I use something similar to this decorator (more or less a thin wrapper around instructor) and have looked a little bit at the codegen + cache route. It gets more interesting with the addition of tool calls, but I've found JSON outputs create quality degradation and reliability issues. My next experiment on that thread is to either use guidance (https://github.com/guidance-ai/guidance) or reimplement some of their heuristics to try to get tool calling without 100% reliance on JSON.
Isn’t that basically just Copilot but way more cumbersome to use?
no https://bsky.app/profile/alternatebuild.dev/post/3lg5a5fq4dc...
Treating it as a compiler is obviously the way right? Setting aside overhead if you’re using local models.. Either the code gen is not deterministic in which case you risk random breakage or it is deterministic and you decided to delete it anyway and punt on ever changing / optimizing it except for in natural language? Why would anyone prefer either case? Code folding works fine if you just don’t want to look at it ever.
I can see this eventually going in the direction of "bidirectional synchronization" of NL representation and code representation (similar to how jupytext allows you work with notebooks in browser or markdown in editor). But a single representation that's completely NL and deliberately throwing away a code representation sounds like it would be the opposite of productivity..
a compiler has one requirement that llms cannot provide. It has to be robust.
Very cool. I implemented something similar for personal use before.
At that time, LLMs weren't as proficient in coding as they are today. Nowadays, the decorator approach might even go further and not just wrap LLM calls but also write Python code based on the description in the Docstring.
This would incentivize writing unambiguous DocStrings, and guarantee (if the LLMs don't hallucinate) consistency between code and documentation.
It would bring us closer to the world that Jensen Huang described, i.e., natural language becoming a programming language.
People have been talking about natural language becoming a programming language for way longer than even Jensen Huang has been talking about it. Once upon a time, they tried to adapt natural language into a programming language, and they came up with this thing called COBOL. Same idea: "then the managers can code, and we won't need to hire so many expensive devs!"
And now the COBOL devs are retiring after a whole career . . .
But isn't it actually more like, COBOL lets you talk in COBOL-ese (which is kinda stilted), whereas LLMs let you talk in LLM-ese (which gets a lot closer to actual language)? And then since the skill cap on language is basically infinite, that this becomes a question of how good you are at saying what you want - to the extent it intersects with what the LLM can do.
COBOL was the best attempt that they could get to in the 1960s. It's the entire reason COBOL has things like paragraphs, things end with periods, etc. They wanted as much of an "English-like syntax" as possible.
The reason it looks so odd today is that so much of modern software is instead the intellectual heir of C.
And yeah, the "skill cap" of describing things is theoretically infinite. My point was this has been tried before and we don't yet know how the actual limitations of an LLM come close to that ideal. People have been trying for decades to describe things in English that still ultimately need to be described in code for them to work; that's why the software industry exists in the first place.
I really like how this integrates with the schema feature I added to the underlying LLM Python library a few weeks ago: https://simonwillison.net/2025/Feb/28/llm-schemas/#using-sch...
Many libraries with the same approach suffer the same flaw: can't easily use the same function with different LLMs at runtime (ie. after importing the module where it is defined).
I initially used the same approach in my library, but changed it to explicitly pass the llm object around and in actual production code it's easier/more flexible to use.
Examples (2nd one also with docstring-based llm query and structured answer): https://github.com/senko/think?tab=readme-ov-file#examples
I often do the reverse -- have LLMs insert docstrings into large, poorly commented codebases that are hard to understand.
Pasting a piece of code into an LLM with the prompt "comment the shit out of this" works quite well.
Matheus Pedroni released a really clever plugin for doing that with LLM the other day: https://mathpn.com/posts/llm-docsmith/
You run it like this:
And it uses a Python concrete syntax tree (with https://pypi.org/project/libcst/) to apply changes to just the docstrings without risk of editing any other code.I was working on a similar thing but for JS.
Imagine this: It would be cool when these functions essentially boiled down to a distilled tiny model just for that functionality instead of an api call to foundation one.
Cool! Looks a lot like Tanuki: https://github.com/Tanuki/tanuki.py
yea its a popular DX at this point: https://blog.alternatebuild.dev/marvin-3x/
Funny. I frequently give the LLM the function and ask it to make the doc string.
TBH I find doc strings very tedious to write. I can see how this would be a great specification for an LLM but I dont know that its actually better than a plain text description of the function since LLMs can handle those just fine and they are easier to write.
There's also promptic which wraps litelm, which supports many, many, many more model providers, and it doesn't even need plugins.
Llm is a cool cli tool, but IMO litellm is a better Python library.
I think LLM's plugin architecture is a better bet for supporting model providers than the way LiteLLM does it.
The problem with LiteLLM's approach is that every model provider needs to be added to the core library - in https://github.com/BerriAI/litellm/tree/main/litellm/llms - and then shipped as a new release.
LLM uses plugins because then there's no need to sync new providers with the core tool. When a new Gemini feature comes out I ship a new release of https://github.com/simonw/llm-gemini - no need for a release of core.
I can wake up one morning and LLM grew support for a bunch of new models overnight because someone else released a plugin.
I'm not saying "LLM is better than LiteLLM" here - LiteLLM is a great library with a whole lot more contributors than LLM, and it's also been fully focused on being a great Python library - LLM has also had more effort invested in the CLI aspect than the Python library aspect so far.
I am confident that a plugin system is a better way to solve this problem generally though.
Is there something like this but for Java?