Your Code is a Prompt

Do we Have to Change how we Code?

Anyone who has done enough development work with LLMs would agree, perhaps after a moment’s hesitation, when I say prompts are code. It’s not procedural code, to be sure, but it has a definite structure to it and contains instructions on how to accomplish a task. If you build an application using prompts, you’re going to check them into your git repository right with the code. Nothing shocking in any of that.

But it might be a bit more shocking when I say the reverse is true:

Your Code is a Prompt

“Code is a prompt” because code is now being used to drive AI. The copilot in your IDE feeds your code to AI to generate more/better code. And the code is going straight into the LLM along with other instructions as part of an overall prompt.

You might argue that your code doesn’t really instruct the LLM to do anything, so it really can’t be prompting it. Wrong! All you are saying is that your code is a poorly designed prompt. If you did a better job of it, the promptness of the code might be more obvious.

So the real question is not “is code a prompt” but “how do I make my code be a decent prompt?” And to answer that, we need to understand how the code can assist LLMs in understanding what it does and why.

LLMs are Junior Developers

LLMs are great at cranking out volumes of code, but not always at cranking out high quality code or completely grasping the requirements. In that regard, they’re like junior developers. And what helps junior developers understand code: comments.

Just any old comments? Of course not. Today we need comments that are specifically tailored to the needs of an LLM.

Let’s look at a very simple (and simplified) example:

Imagine you want a command line tool that will tell you all the files in a directory (and its subdirectory) that are over a certain size. You want to pipe the output of this command into some other command (don’t care what) so the output needs to be clean.

You’ve written this shell of a program and would like to have ChatGPT finish it:

import os
import sys

def find_large_files(path, size):
    # TBD
    pass

if __name__ == "__main__":
    # get arguments to program, the path and the size
    if len(sys.argv) != 3:
        print("Usage: python main.py <path> <size>")
        sys.exit(1)
    path = sys.argv[1]
    size = sys.argv[2]
    find_large_files(path, size)

So you write “Complete this code:” in the ChatGPT window and then paste in the code below that.

Here’s the full output:

import os
import sys

def find_large_files(path, size):
    try:
        size = int(size)  # Convert size to an integer
    except ValueError:
        print("Error: Size must be an integer.")
        sys.exit(1)

    if not os.path.exists(path):
        print(f"Error: Path '{path}' does not exist.")
        sys.exit(1)

    print(f"Searching for files larger than {size} bytes in '{path}'...")

    for root, dirs, files in os.walk(path):
        for file in files:
            file_path = os.path.join(root, file)
            try:
                file_size = os.path.getsize(file_path)
                if file_size > size:
                    print(f"{file_path} ({file_size} bytes)")
            except (OSError, FileNotFoundError) as e:
                print(f"Could not access file: {file_path}. Error: {e}")

if __name__ == "__main__":
    # Get arguments to program, the path, and the size
    if len(sys.argv) != 3:
        print("Usage: python main.py <path> <size>")
        sys.exit(1)
    path = sys.argv[1]
    size = sys.argv[2]
    find_large_files(path, size)

It’s done a decent job of figuring out what we want, but it’s not quite right. It’s adding all sort of extraneous outputs to the list of file names, which we don’t want. So let’s put a prompt inside the code to explain what we want:

"""
This is a utility that outputs the full paths and names of all
files in and under a directory in that are over a certain size.
The intent is to be able to pipe the output of this program to
another program for some other purpose (that is up to the user).

Any output which is not part of the full path of a file should
be printed to standard error to keep standard output clean.

IMPORTANT: Retain this comment at the top of the file
"""
import os
import sys

def find_large_files(path, size):
    # TBD
    pass

if __name__ == "__main__":
    # get arguments to program, the path and the size
    if len(sys.argv) != 3:
        print("Usage: python main.py <path> <size>")
        sys.exit(1)
    path = sys.argv[1]
    size = sys.argv[2]
    find_large_files(path, size)

And that generates:

"""
This is a utility that outputs the full paths and names of all
files in and under a directory that are over a certain size.
The intent is to be able to pipe the output of this program to
another program for some other purpose (that is up to the user).

Any output which is not part of the full path of a file should
be printed to standard error to keep standard output clean.

IMPORTANT: Retain this comment at the top of the file
"""
import os
import sys

def find_large_files(path, size):
    try:
        size = int(size)  # Ensure size is an integer
    except ValueError:
        print(f"Error: Size '{size}' must be an integer.", file=sys.stderr)
        sys.exit(1)

    if not os.path.exists(path):
        print(f"Error: Path '{path}' does not exist.", file=sys.stderr)
        sys.exit(1)

    if not os.path.isdir(path):
        print(f"Error: Path '{path}' is not a directory.", file=sys.stderr)
        sys.exit(1)

    for root, _, files in os.walk(path):
        for file in files:
            full_path = os.path.join(root, file)
            try:
                file_size = os.path.getsize(full_path)
                if file_size > size:
                    print(full_path)  # Output full path to stdout
            except OSError as e:
                print(f"Error accessing file '{full_path}': {e}", file=sys.stderr)

if __name__ == "__main__":
    # Get arguments to program, the path and the size
    if len(sys.argv) != 3:
        print("Usage: python main.py <path> <size>", file=sys.stderr)
        sys.exit(1)

    path = sys.argv[1]
    size = sys.argv[2]
    find_large_files(path, size)

Now the code is correct because we’ve added prompts into the code. Could we have added the prompt outside of the code? Yes. But then the code-specific prompt wouldn’t be associated with the specific code and would be just ephemeral. It’s easy-peasy to keep it around by putting it in the code.¹

You might think that once we’ve generated all the code we don’t need this extra prompt-comment. But you would be wrong: it works well for future editing too. Let’s say I modify the comment at the top to change the output from just file names to sizes and filenames:

"""
This is a utility that outputs the sizes (without commas), 
a space, and then full paths and names of all files in and 
under a directory that are over a certain size.

The intent is to be able to pipe the output of this program to
another program for some other purpose (e.g., sort the files by size).

Any output which is not part of the full path of a file should
be printed to standard error to keep standard output clean.

IMPORTANT: Retain this comment at the top of the file
"""

(The bolding is not in the actual code)

Pass in to ChatGPT the directive “Please update the code to match its definition:” followed by the source code (with updated comment), and it updates the code as we requested:

file_size = os.path.getsize(full_path)
if file_size > size:
    print(f"{file_size} {full_path}") 

Now “prompts are code” has a second meaning!

What Should We Do?

Adding comments is time consuming, which is why they’re so rare.² When and where is it worth it?

My advice to you is to experiment with this. Code that you’re developing now, put in some prompt-comments that describe the non-obvious aspects of the code. It should make your Copilot better at helping you with the code. It may be very hard to tell (unless you do A/B experiments like above). But I think that, over time, you’ll find that a bit of extra hints for the LLM go a long way.³

1. Well, not completely easy-peasy: do you know what happens if we omit the line “IMPORTANT: Retain this comment at the top of the file”? Yep, it removes the comment.

2. And also why so many developers believe the myth of “self-documenting code”: it’s a rationalization for their being lazy.

3. Comments also make it easier for human readers to read the code, but since that’s clearly insufficient motivation for people to spend the time it takes to write them, “do it for AI” is where I’ll hang this hat.