Hi all. I’ve been building my own little fediverse setup at dboft.com to fit how I like to read, and I’m grateful for all the work and community that makes something like this possible. Looking forward to being here with you all!

  • Otter@lemmy.caEnglish
    2·
    4 hours ago

    That’s very cool, and I appreciate the thought that you’ve put into this, including the naming.

    How does the embedding side change as it scales up? (Storage costs and performance). Brainstorming based on your description, I assume the embedding process is the most resource intensive, and lookups are basically free? If so, would the limit be how fast it can process all new posts?

    I wonder if it would be helpful to run a threadiverse search service like this, which other instances can plug in to. Theres no strong reason for everyone to run their own, since they can just filter out any results that a particular instance has blocked or banned. In our case, we wouldn’t want to replicate that work across the few instances that we run.

    • kalman@dboft.comOP
      3·
      4 hours ago

      Pretty much, yes. Embedding a post takes about half a second on dboft’s small server, and the lookup afterwards is under a millisecond. At 3–4 thousand new posts and comments a day that’s maybe 40 minutes of work, so the ceiling is exactly what you guessed: how fast new posts can be embedded. More cores or a small GPU would push it way out i would think. Storage grows too, about 14 KB per post with its index, so 35k today is ~500 MB and a million would be ~14 GB. It all lives on disk in Postgres; keeping the index in RAM just makes lookups sub-millisecond instead of a few milliseconds, and Postgres caches the busy parts on its own, so it degrades gently rather than falling over. This was just an idea; I’m just putting it into together to see if it holds up. I like the idea of a shared service… I wonder what the memory constraints would be at scale.