Skip to main content

AI Skill System with Hot-Reload Prompts

You run an AI agent platform. Each "skill" is a markdown file with a system prompt, some examples, and configuration. You have 300 of these files. On every request, the agent loads the relevant skill, assembles the prompt, and sends it to an LLM.

Reading 300 files from disk on every request is wasteful. Even reading one file from disk every request is slower than it needs to be.

With inhouse, you load skills once and keep them hot. When you edit a skill file in your editor, the library notices the file modification time changed and invalidates that single entry on the next read.

The setup​

Store all your skills in a directory. Each file is a markdown document with frontmatter for metadata and a body for the system prompt.

skills/
  customer-support.md
  code-review.md
  data-analysis.md
  translation.md
  ...

The loader reads a file and returns its contents. The decorator watches for file changes and invalidates automatically.

from inhouse.files import file_cache

SKILLS_DIR = Path("skills/")

@file_cache(ttl_seconds=3600, watch_files=True)
def load_skill(path: str) -> str:
    full_path = SKILLS_DIR / path
    return full_path.read_text(encoding="utf-8")

Pass watch_files=True and inhouse walks your function arguments looking for file paths. When it finds one, it snapshots the file's mtime at write time. On cache hit, it checks if the file changed. If it did, the entry is deleted and recomputed on the next call.

Preloading at startup​

For the 300 skills you know about at boot, warm the cache so the first request never pays a cold-start penalty:

def warm_skill_cache():
    for skill_file in SKILLS_DIR.glob("*.md"):
        load_skill(skill_file.name)

This populates the cache with all 300 skills. The first user request for any skill hits cache immediately. And because watch_files=True is set, editing any file invalidates just that skill.

Assembling a prompt without re-reading disk​

@file_cache(ttl_seconds=3600, watch_files=["*.md"])
def build_system_prompt(skill_name: str, user_context: dict) -> str:
    skill_content = load_skill(f"{skill_name}.md")
    personality = user_context.get("tone", "professional")
    return f"""{skill_content}

---

Tone: {personality}
Current time: {datetime.now().isoformat()}
User language: {user_context.get("locale", "en")}
"""

The skill file is cached. The assembled system prompt is cached separately. Edit the skill markdown and both caches invalidate because watch_files detects the mtime change on the underlying .md file.

Disabling the cache during development​

While you are iterating on prompts, turn the cache off entirely:

INHOUSE_CACHE_DISABLE=1 uvicorn app:main

Or call disable_all() from a debug endpoint. Every cache check short-circuits, and your functions run fresh every time. When you are done, enable_all() turns it back on. No code changes, no restart needed for the toggle.

Why this works​

  • One mtime stat call per check. Cheap enough for skill loading, but you would not want it on a hot request path serving thousands of requests per second.
  • The long TTL of 3600 seconds means skills stay warm for an hour. If nobody requests a skill for an hour, it expires naturally.
  • watch_files=["*.md"] filters only markdown files so other string arguments don't trigger file checks.
  • Preloading at startup means zero cold-start latency for known skills.

Alternative: cache at startup and never watch​

If you prefer to load everything once at startup and never watch for changes (restart to pick up edits), skip file watching entirely:

SKILL_CACHE: dict[str, str] = {}

def load_all_skills():
    for f in SKILLS_DIR.glob("*.md"):
        SKILL_CACHE[f.stem] = f.read_text(encoding="utf-8")

load_all_skills()

def get_skill(name: str) -> str:
    return SKILL_CACHE[name]

Simple. No TTLs. No file watching. Restart the server to pick up changes. If that fits your workflow, it is hard to beat.