Anyone want to help me design a rough algorithm for re-crawling the Twtxt network for the Yarns (Yarn Search) engine hosted at search.twtxt.net – This is the only experience I have with building search engines, and it would be good to bounce ideas off you smart folks 🙃 – How often should it recrawl? How often should it rescrape a feed? etc…