R
rfc9309

Any
Batchfile
Blade
C
C#
C++
CMake
CSS
Dockerfile
Go
HCL
HTML
Java
JavaScript
Kotlin
Makefile
Objective-C
PHP
Python
Ruby
SCSS
Shell
Swift
TSX
TypeScript
Vue

Projects with this topic

Sort by:

Sort by
Updated date
Name
Name, descending
Oldest updated
Oldest created
Last created
Most stars
Hide archived projects
Show archived projects
Show archived projects only

View robotstxtr project

Bart Turczynski / robotstxtr

Engine-aware robots.txt policy for R with Google's matcher algorithm.

R rstats r-package robots-txt robots-exclu... rfc9309 crawler web-crawling googlebot seo technical-seo

0

Updated Sep 30, 2026

0 0 0 0

Updated Sep 30, 2026