milknet/pages
1
0
Fork 0
mirror of https://github.com/milk-net/milk-net.github.io.git synced 2025-04-16 07:33:42 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

Create robots.txt

Browse source
This commit is contained in:
Voxel 2025-04-14 10:45:00 -04:00 • committed by GitHub
parent 685b6ea8c5
commit 7c7ba32314
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 59 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

59
robots.txt Normal file
View file

@ -0,0 +1,59 @@
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: Sogou
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: wget
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: WebEMailExtractor
Disallow: /
User-agent: EmailHarvester
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: EmailSucker
Disallow: /
User-agent: EmailCrawler
Disallow: /