30:0e84a4a22acd
Anton Shestakov <av6@dwimlabs.net>, Sat, 09 Jun 2018 15:14:44 +0800
robots: disallow some non-essential pages

roles/hgweb/files/robots.txt

Permissions: -rw-r--r--

File Latest Revisions Annotate Diff Comparison
Other formats:
JSON Raw
Feeds:
Atom RSS 
User-agent: *
# Expensive pages
Disallow: /*/annotate/
Disallow: /*/comparison/
Disallow: /*/diff/
Disallow: /*/archive/
# File log is too much
Disallow: /*/log/*/
# API is off-limits
Disallow: /*/json-*
Disallow: /*/raw-*
# Paginate instead
Disallow: /*?revcount=
# Graph page is a more expensive log
Disallow: /*/graph$
Disallow: /*/graph/
# It's for humans
Disallow: /*/help$
Disallow: /*/help/