In Firefox at least, Stylish isn't actually what supplies the per-site logic, to my knowledge. That's actually done by the @-moz-document at-rule, which is specific to Gecko. You could use it in a user stylesheet without Stylish and it'd work fine.

However, Stylish is such an awesome frontend and it makes it so easy to do this sort of thing that I wouldn't do it any other way, myself!

(The above explanation probably doesn't apply to the other browsers that Stylish works on, though.)