Anyone want to help me design a rough algorithm for re-crawling the Twtxt network for the Yarns (Yarn Search) engine hosted at search.twtxt.net โ This is the only experience I have with building search engines, and it would be good to bounce ideas off you smart folks ๐ โ How often should it recrawl? How often should it rescrape a feed? etcโฆ