User-agent: *
Crawl-delay: 100
Disallow: /auth
Disallow: /search

Sitemap: https://data.gov.au/sitemap.xml