Refactor cache control settings and improve robots.txt handling; update definition retrieval to synchronous method

2026-05-17 21:45:59 -07:00
parent 4a4b759a37
commit 6b028fdbda
8 changed files with 45 additions and 27 deletions
@@ -1,4 +0,0 @@
-# allow crawling everything by default
-User-agent: *
-Disallow:
-Sitemap: https://howdoyouconvert.com/sitemap.xml