Documentation
¶
Overview ¶
Batch process robots.txt files from archives like https://github.com/nrjones8/robots-dot-txt-archive-bot/tree/master/data/cleaned into Anubis CEL policies. Usage: go run batch_process.go <directory with robots.txt files>
Click to show internal directories.
Click to hide internal directories.