.htaccess, prevent hotlinking, allow big bots, allow some access and allow my own domain without adding it statically

Question

1. I want to allow image crawling on my site from a couple of different bots and exclude all others.
2. I want to allow images in at least one folder to not be blocked for any request.
3. I don't want to block image requests from visitors on my own site.
4. I don't want to include my domain name in the .htaccess file for portability.

The reason I ask this here and don't simply test the following code myself is that I work on my own and have no colleges to ask or external resources to test from. I think what I've got is correct but I find .htaccess rules extremely confusing, and I don't know what I don't even know at this point.

RewriteCond %{HTTP_REFERER} !^$ [OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?bing\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?facebook\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?google\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?instagram\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?linkedin\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?reddit\..+$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?twitter\..+$ [NC,OR]
RewriteCond %{REQUEST_URI} !^/cross-origin-resources/  [NC,OR]
RewriteCond %{HTTP_HOST}@@%{HTTP_REFERER} !^([^@]*)@@https?://\1/.* [NC]
RewriteRule \.(bmp|gif|jpe?g|png|webp)$ - [F,L,NC]

I've tested it on htaccess tester, and looks good but does complain about the second last line when tested using the following URL: http://www.example.co.uk/poignant/foo.webp

Answer 1

You have the logic in reverse. As written these conditions (RewriteCond directives) will always be successful and the request will always be blocked.

You have a series of negated conditions that are OR'd. These would only fail (ie. not block the request) if all the conditions match, which is impossible. (eg. The Referer header cannot be bing and facebook.)

You need to remove the OR flag on all your RewriteCond directives, so they are implicitly AND'd.

Incidentally, the suggestion in comments from @StephenOstermiller to combine the HTTP_REFERER checks into one (which is a good one) is the equivalent to having the individual conditions AND'd, not OR'd (as you have posted initially).

1. I want to allow image crawling on my site from a couple of different bots and exclude all others.

Once you've corrected the OR/AND as stated above, this rule will likely allow ALL bots to crawl your site images because bots generally do not send a Referer header. These directives are not really about "crawling", they allow certain websites to display your images on their domain (ie. hotlinking). This is probably the intention, however, it's not what you are stating in point #1.

(To block bots from crawling your site you would need to check the User-Agent request header, ie. HTTP_USER_AGENT - which would probably be better done in a separate rule.)

RewriteCond %{HTTP_REFERER} !^https?://(www\.)?bing\..+$

Minor point, but the +$ at the end of the regex is superfluous. There's no need to match the entire Referer when you are only interested in the hostname. Although these sites probably have a Referrer-Policy set that prevents the URL-path being sent (by the browser) in the Referer header anyway, but it is still unnecessary.

RewriteCond %{HTTP_HOST}@@%{HTTP_REFERER} !^([^@]*)@@https?://\1/.* [NC]

In comments, you were asking what this line does. This satisfies points #3 and #4 in your list, so it is certainly needed. It ensures that the requested Host header (HTTP_HOST) matches the hostname in the Referer. So the request is coming from the same site.

The alternative is to hardcode your domain in the condition, which you are trying to avoid.

(Again, the trailing .* on the regex is unnecessary and should be removed.)

This is achieved by using an internal backreference \1 in the regex against the HTTP_REFERER that matches HTTP_HOST in the TestString (first argument). The @@ string is just an arbitrary string that does not occur in the HTTP_HOST or HTTP_REFERER server variables.

This is clearer if you expand the TestString to see what is being matched. For example, if you make an internal request to https://example.com/myimage.jpg from your homepage (ie. https://example.com/) then the TestString in the RewriteCond directive is:

example.com@@https://example.com/

This is then matched against the regex ^([^@]*)@@https?://\1/ (the ! prefix on the CondPattern is an operator and is part of the argument, not the regex).

1. ([^@]*) - the first capturing group captures example.com (The value of HTTP_HOST).
2. @@https?:// - simply matches @@https:// in the TestString (part of the HTTP_REFERER).
3. \1 - this is an internal backreference. So this must match the value captured from the first capturing group (#1 above). In this example, it must match example.com. And it does, so there is a successful match.
4. The ! prefix on the CondPattern (not strictly part of the regex), negates the whole expression, so the condition is successful when the regex does not match.

So, in the above example, the regex matches and so the condition fails (because it's negated), so the rule is not triggered and the request is not blocked.

However, if a request is made to https://example.com/myimage.jpg from an external site, eg. https://external-site.example/ then the TestString in the RewriteCond directive is:

example.com@@https://external-site.example/

Following the steps above, the regex fails to match (because external-site.example does not match example.com). The negated condition is therefore successful and the rule is triggered, so the request is blocked. (Unless one of the other conditions failed.)

Note that with the condition as written, www.example.com is different to example.com. For example, if you were on example.com and you used an absolute URL to your image using www.example.com then the regex will fail to match and the request will be blocked. This could perhaps be incorporated into the regex, to allow for this. But this is very much an edge case and can be avoided with a canonical 301 redirect earlier in the config.

RewriteCond %{HTTP_REFERER} !^$

This allows an empty (or not present) Referer header. You "probably" do need this. It allows bots to crawl your images. It permits direct requests to images. It also allows users who have chosen to suppress the Referer header to be able to view your images on your site.

HOWEVER, it's also possible these days for a site to set a Referrer-Policy that completely suppresses the Referer header being sent (by the browser) and so bypasses your hotlink protection.

RewriteRule \.(bmp|gif|jpe?g|png|webp)$ - [F,L,NC]

Minor point, but the L flag is not required when the F flag is used (it is implied).

Are you really serving .bmp images?!

Aside: Sites don't necessarily "hotlink"

Some of these external sites (bing, Facebook, Google, Instagram, LinkedIn, Reddit, twitter, etc.) don't necessarily "hotlink" images anyway. They often make their own (resized/compressed) "copy" of the image instead (a bot makes the initial request to retrieve the image - with no Referer - so the request is not blocked).

So, explicitly permitting some of these sites in your "hotlink-protection" script might not be necessary anyway.

Summary

Taking the above points into consideration, the directives should look more like this:

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?(bing|facebook|google|instagram|linkedin|reddit|twitter)\.
RewriteCond %{REQUEST_URI} !^/cross-origin-resources/  [NC]
RewriteCond %{HTTP_HOST}@@%{HTTP_REFERER} !^([^@]*)@@https?://\1/
RewriteRule \.(gif|jpe?g|png|webp|bmp)$ - [F,NC]