twtxt.net prologic@ "Wait... So you actually wrote a more elaborate crawler without taking a shortcut like I did using colly (not that it really helps much) Hmmm? � ..."

Tue, Jan 12 10:37 2021 (3y ago)

↳ In-reply-to » I just built a poc search engine / crawler for Twtxt. I managed to crawl this pod (twtxt.net) and a couple of others (sorry @etux and @xuu I used your pods in the tests too!). So far so good. I might keep going with this and see what happens 😀

Wait… So you actually wrote a more elaborate crawler without taking a shortcut like I did using colly (not that it really helps much) Hmmm? 🤔 Can we take it a bit further, make a daemon/server out of it, a web interface to search what it crawls using bleve and some tools (API, Web UI) to let people add more “feeds” to crawl? 🤔

⤋ Read More