Currently everything on the Internet is assumed to be free. Robots.txt is just a suggestion and not legally enforceable. I assume RSL is supposed to communicate terms of use explicitly, like a EULA.

Robots is just a suggestion and so is this because scaraper never cared about legality of things. All this thing does is make license more easily accessible but consequently, do we want to make it easy for them in the first place? Make scrapers work for it.