From f9b94e3ba677fd19c311fdde7cd9da8709953000 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Sat, 5 Dec 2020 17:04:13 -0300 Subject: Feat: adds a robots.txt Robots tend to crawl all tag combinations, which make a huge performance impact in larger datasets, specially regarding database queries. This robots.txt file will prevent that regular search engines to index the application, which is already a search index. --- www/robots.txt | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 www/robots.txt diff --git a/www/robots.txt b/www/robots.txt new file mode 100644 index 0000000..1f53798 --- /dev/null +++ b/www/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: / -- cgit v1.2.3