diff options
| author | Silvio Rhatto <rhatto@riseup.net> | 2020-12-05 17:04:13 -0300 |
|---|---|---|
| committer | Silvio Rhatto <rhatto@riseup.net> | 2020-12-05 17:04:13 -0300 |
| commit | f9b94e3ba677fd19c311fdde7cd9da8709953000 (patch) | |
| tree | 6d95d3bebd6c908368c105af3bc279f340ea3b68 | |
| parent | 37f10bd9d3f955fd648424d3908d23c33bea39fc (diff) | |
| download | semanticscuttle-f9b94e3ba677fd19c311fdde7cd9da8709953000.tar.gz semanticscuttle-f9b94e3ba677fd19c311fdde7cd9da8709953000.tar.bz2 | |
Feat: adds a robots.txt
Robots tend to crawl all tag combinations, which make a huge performance
impact in larger datasets, specially regarding database queries.
This robots.txt file will prevent that regular search engines to
index the application, which is already a search index.
| -rw-r--r-- | www/robots.txt | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/www/robots.txt b/www/robots.txt new file mode 100644 index 0000000..1f53798 --- /dev/null +++ b/www/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: / |