robots.txt Fetch Frequency: What Historical Logs Can and Cannot Prove
A saved crawler-log aggregation reveals two incorrect averages and limits in the original method. Counts, successful rule retrieval and the time a change takes effect are different measurements.










