What tools and projects do you use for your local LLM chats and agents?

I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

  • aihostbox@programming.devEnglish
    1·
    1 day ago

    What’s stuck for me after trying too many things:

    • Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
    • Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
    • LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
    • SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it

    Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.