robots.txt code snippets in typescript

robots.txt validator: checks if a website's robots.txt file is valid. in typescript