From: Mary Umoh <umohm12@gmail.com>
Date: Thu, 24 Aug 2017 01:03:12 +0000 (-0400)
Subject: Bug 1393145 - See if we can add a directive/rule to robots.txt to get search engines... 
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=1feabf50777a6f0f4193f467faad9f996e729367;p=thirdparty%2Fbugzilla.git

Bug 1393145 - See if we can add a directive/rule to robots.txt to get search engines to crawl more slowly
---

diff --git a/template/en/default/robots.txt.tmpl b/template/en/default/robots.txt.tmpl
index d8c1b5b86..c4948efe5 100644
--- a/template/en/default/robots.txt.tmpl
+++ b/template/en/default/robots.txt.tmpl
@@ -1,5 +1,6 @@
 User-agent: *
 Disallow: /
+Crawl-delay: 30
 
 [% IF NOT urlbase.matches("bugzilla-dev") %]