diff options
| author | Silvio Rhatto <rhatto@riseup.net> | 2020-12-05 17:04:13 -0300 |
|---|---|---|
| committer | Silvio Rhatto <rhatto@riseup.net> | 2020-12-05 17:04:13 -0300 |
| commit | f9b94e3ba677fd19c311fdde7cd9da8709953000 (patch) | |
| tree | 6d95d3bebd6c908368c105af3bc279f340ea3b68 /data | |
| parent | 37f10bd9d3f955fd648424d3908d23c33bea39fc (diff) | |
| download | semanticscuttle-f9b94e3ba677fd19c311fdde7cd9da8709953000.tar.gz semanticscuttle-f9b94e3ba677fd19c311fdde7cd9da8709953000.tar.bz2 | |
Feat: adds a robots.txt
Robots tend to crawl all tag combinations, which make a huge performance
impact in larger datasets, specially regarding database queries.
This robots.txt file will prevent that regular search engines to
index the application, which is already a search index.
Diffstat (limited to 'data')
0 files changed, 0 insertions, 0 deletions