For context I created a video search engine last year, I shut it down and put the data online. You can read about it here: https://www.bendangelo.me/2024/07/16/failed-attempt-at-creating-a-video-search-engine/
I put that project on hold because of scaling issues, anyway I’m back with an other idea. I’ve been frustrated with how AI slop is ruining the internet and recently it’s been hitting Youitube pretty hard with AI videos. I’m brainstorming a tool for people to selfhost:
Self-hosted crawler: Pick which sites/videos to index (blogs, forums, YT channels, etc.). AI chat interface: Ask questions like, “Show me Rust tutorials from 2023” or “Summarize recent posts about homelab backups.” Optional sharing: Pool indexes with trusted friends/communities.
Why? No Google/YouTube spam—only content you choose. Works offline (archive forums, videos, docs). Local AI (Mistral) or cloud (paid) for smarter searches.
Would this be useful to you? What sites would you crawl? Any killer features I’m missing?
Prototype in progress—just testing interest!
Not really. I could use some good selfhosted search engine. I mean all the existing projects (which is just YaCy, to my knowledge) are a bit dated. Nowadays we only got metasearch engines and we’re relying on Google, Bing etc.
But I don’t need any chatbot enhancements. That’s usually something I skip when using Google or Bing because it doesn’t work well. The AI summaries tend to be wrong, and it’s bad at looking up niche information, which is something I need a search engine to be able to find. The AI just cites the most common slop, or at best the Wikipedia article. But I don’t really need any fancy software to get there… So for me, we don’t need any AI augmentation.
And I think the old way of googling was fine. Just teach people to put in the words that are likely to be in the article they want to find. That’d be something like “Rust new features 2023” or “homelab backup blog”. Sure you can strap on a chatbot and put in entire natural language questions. But I think that’s completely unnecessary. We have brains and we’re perfectly able to translate our questions into search queries with little effort… If somebody teches us what to type into the search bar, and why.