Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or register to comment.
- 96gb ddr5
- intel i7 12th gen
- RTX 3060
it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.
I run too many models to count really. everything through CLI because I’m dissatisfied with all the sloppy UIs.
96gb ddr5
Look at the billionaire flaunting their wealth
What a timely comment.
I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB
Combined with a 5600x3d and 32gb of system memory.
I just started messing with local LLMs yesterday. “Getting started” is rather easy.
pacman -S lm-studio(I use arch, btw), launch it, download a model that your hardware supports, and start talking.Right now I’m working on getting it to talk to vscode.
32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it’s a fine tune of Qwen 3.6) with Zoo Code
24GB GPU
Qwen 3.8 ( 2 variants )
Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab
Zoo Code>OpenWebUI>lama.cpp for coding , mostly scripts and html
Qwen 3.8 was a game changer for me.
If you haven’t seen it already, take a look at !localllama@sh.itjust.works
I recently gave Strata a shot for my local AI. I’m very impressed with the performance they have eeked out of my hardware. I’ve got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.
Strata acts like an OpenAI compatible API server, I then have Hermes as a “harness” to make an “agent”, which I interact with on “buzz”, which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.
If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.
One thing I didn’t mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn’t a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn’t have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.
I self-host my own neural network, all within the confines of my own skull.
That 33% downtime though
The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).
It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.
Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
It’s better than github
I’m just glad that this is both Top & Hot comment. 🧠✊
How much RAM is it using?
Made me think of Johnny Mnemonic. https://www.youtube.com/watch?v=jS2GFqT5ycU
Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it’ll need. Maybe Johnny’s world is just that advanced lol
Not me. Call me old-fashioned, but I prefer organic.
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.
I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/
I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.
I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.
Would be interested to know your experience with fcli if you’d share more. Just hearing about it.
I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅
In essence, the problem is twofold:
- LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
- LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.
So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.
As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.
Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.
The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.
There’s also a creds uck to make your secrets more portable and (a bit) safer.
In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)
Yes, were all on c/LocalLLaMa
I’m running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.
Is that the only community? I see the last activity was 2 months ago.
I see the last activity was 2 months ago.
that means no one from your instance is subscribed, if you subscribe then you’ll get new posts
you can see that there are new posts here https://sh.itjust.works/c/localllama
Posts from 16 minutes and 2 days ago here. Maybe your instance is not federating with theirs.
oh yea feddit blocks shitjustworks https://feddit.online/instance/sh.itjust.works/posts
maybe @xSikes@feddit.online could ask their admins to convert the defederation to a “silenced” or whatever that feature is on PieFed
I’m self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.
(Prob not the answer you were looking for, but a reminder that “AI” is more than just LLMs.)
Does Immich sell/use/keep your ML-based FR? Or is this totally local a never leaves your instance?
That was always the question that bordered me… I run Leeche its a very simple and minimal photo album service and I really like it ! But my family want something that’s more like immich but I’m very cautious about what i’m selfhosting and something that collects data points on my behalf is a no-go !
Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it’s ridiculously good software).
I do not run Immich but their privacy policy states:
Locally Stored Data: Immich stores all your photos, albums, settings, and other metadata locally on your device. We do not have access to this data, nor do we transmit or store it on any of our servers.
https://immich.app/privacy-policy
Additionally in their docs I can find no mention of connecting to outside services. I do see references to downloading hugging face models.
https://docs.immich.app/features/facial-recognition#how-face-detection-works
Thank you very much ! I didn’t expected someone doing my part to RTFM o_O !
Without doing again the heavy reading part, what should I expect from hugging face models?
If my reading skills are correct it should be local, but I’m really paranoid about this stuff ^^ Even though I read your posted links, I’m still not sure if I should trust them…
Something written in a manual and what the code realy does are sometimes far appart ^^’
As a service? No, not really.
I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.
They aren’t great for search IMO, and search summaries can be quite bad.
I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.
Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren’t good at everything. They might claim it, but the reality is a tailored model is often the better approach.
Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.
A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren’t great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.
Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.
- Qwen3.6 35b a3b Q4_K_M running on llama.cpp
- Open-WebUI as a frontend
- integrations with an embedding model and searxng & crawl4ai
- doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc
Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
What’s your rig?
I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.
Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.
Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday
Yah doing similar on 5060ti.
Hardware is a 5070 12GB VRAM, an AMD 5950x and 64G of DDR4 running on Nobara (Fedora spinoff).
I use 3 models for most things: Ternary Bonsai 27b and Gemma 4 E4B Uncensored Q8_0 for everyday stuff, and Qwen Agentworld 35B A3B Q8_0 with CPU offloading for tasks that require to keep the model on track for long periods of time.
Everyday performance is best with Bonsai for me - it comfortably fits in my VRAM and is noticeably better with staying on task than similarily sized models. But if i want it to work overnight on a task without issue, the lower speed of the agentic Qwen is outweighed by its capability to do this.
They are allowed tool calls for websearch and reading webpages and have a sandboxed directory for R/W operations, but anything regarding command line and R/W access in other areas has to be manually authorized by me (i’m not willing to play with fire here).
They are currently not reachable from outside, but i am currently building a smaller machine as a home server, which will then be using a 3070Ti and 32G RAM to provide probably the Bonsai Model for me on the go per VPN.
The smaller models are here for quickly creating documents i do not care to do myself, RAG for technical documentation, creating quick python scripts for automating small tasks and teaching me RegEx and Python(mostly by analyzing the LLM output and playing around with it) and also some RP as NPC in a fantasy setting. The agentic model is currently working on cataloguing my games collection (around 36000 titles) by looking up games, their genres and descriptions and filling in blanks in the database. (Funnily, this task was refused by any model freely available because of the size of the database. Even the local models moan and bitch that it takes so long. No shit, this is why i outsource this!)
My server’s old GPU (8gb) can’t run anything useful LLW wise I don’t think.
Have run some stuff locally on my main computer, but nothing setup as a proper service.
Not for complex tasks, really. Gemma4b or similar might be just enough as the backend for a voice assistant replacing Alexa (which I honestly hope, I need to get family members off of that fucking spy devices…). Small web searches might just fit. Or image recognition for Nextcloud Memories, stuff like that.
