The vastness of the web imposes a prohibitive cost on building large-scale search engines with limited resources. Crawl frontiers thus need to be optimized to improve the coverage and freshness of crawled content. In this paper, we propose an approach for modeling the dynamics of change in the web using archived copies of webpages. To evaluate its utility, we conduct a preliminary study on the scholarly web using 19,977 seed URLs of authors' homepages obtained from their Google Scholar profiles. We first obtain archived copies of these webpages from the Internet Archive (IA), and estimate when their actual updates occurred. Next, we apply maximum likelihood to estimate their mean update frequency ($\lambda$) values. Our evaluation shows that $\lambda$ values derived from a short history of archived data provide a good estimate for the true update frequency in the short-term, and that our method provides better estimations of updates at a fraction of resources compared to the baseline models. Based on this, we demonstrate the utility of archived data to optimize the crawling strategy of web crawlers, and uncover important challenges that inspire future research directions.

中文翻译：

---

使用存档数据为学术网页更新建模

网络的广阔性给构建具有有限资源的大型搜索引擎带来了高昂的成本。因此，需要优化爬网边界以提高爬网内容的覆盖范围和新鲜度。在本文中，我们提出了一种使用网页的存档副本对网络变化动态进行建模的方法。为了评估其效用，我们使用从其Google学术搜索个人资料获得的作者主页的19,977个种子URL在学术网站上进行了初步研究。我们首先从Internet存档（IA）获取这些网页的存档副本，并估计它们的实际更新时间。接下来，我们应用最大似然估计其平均更新频率（$ \ lambda $）值。我们的评估表明，从已存档数据的短历史记录中得出的$ \ lambda $值可以很好地估算短期内的真实更新频率，并且与基线相比，我们的方法可以在较少资源的情况下提供更好的更新估算楷模。在此基础上，我们演示了归档数据的实用程序，可以优化Web爬网程序的爬网策略，并发现可以激发未来研究方向的重要挑战。