How to unblock certain User-Agent's on a Lightsail Bitnami Wordpress instance

Question

I'm currently running Wordpress on Lightsail using a Bitnami distribution. In attempting to run the Broken Link Checker Plugin (which is a cloud service that crawls the site looking for broken links), it is receiving HTTP/500 errors, causing it to abort the link checking.

In contacting the BLC devs, they state that it is due to the fact that:
>seems you are using CloudFront and might be blocking our BLC bot, in such cases, you might need to whitelist our UA, and our IPs

Further, BLC devs state that they can see I'm using Cloudfront by running cURL against my site:

```
curl -IL https://adventuretaco.com/go/if-everybody-did-jo-anne-stover/
```
This command does return headers that imply some Cloudfront interaction, but my assumption is that Lightsail is just built on some Cloudfront infrastructure?

To my knowledge, I'm only using Lightsail and not Cloudfront at all. Regardless, I'd like to configure my instance to allow the User-Agent from the Broken Link Checker (`WPMU DEV Broken Link Checker Spider`). I'm not sure how to do that at all, so help would be appreciated.

Riku_Kobayashi · Accepted Answer

Hello.

When you run the curl command described, a redirect occurs and it appears that CloudFront is being used at the redirect destination.  
So, I don't think the Lightsail instance (adventuretaco.com) itself uses CloudFront.  
It's probably blocked by the redirect destination "www.amazon.com," so I think it's a good idea to exclude the URL from being checked for broken links.  
```
curl -IL https://adventuretaco.com/go/if-everybody-did-jo-anne-stover/
HTTP/2 302 
x-robots-tag: noindex, nofollow
x-redirect-by: WordPress
location: https://amzn.to/3tvpTdU
cache-control: max-age=0
expires: Tue, 04 Jun 2024 00:16:01 GMT
vary: Accept-Encoding
content-type: text/html; charset=UTF-8
date: Tue, 04 Jun 2024 00:16:01 GMT
server: Apache

HTTP/2 301 
cache-control: private, max-age=90
content-security-policy: referrer always;
content-type: text/html; charset=utf-8
date: Tue, 04 Jun 2024 00:16:02 GMT
location: https://www.amazon.com/If-Everybody-Did-Ann-Stover/dp/0890844879?crid=29R42WY6XBCF3&dchild=1&keywords=everybody+did&qid=1612561014&sprefix=everybody+did,aps,364&sr=8-2&linkCode=sl1&tag=srchamzn-20&linkId=ab58ffd173d8b3808625c3ed5343cf51&language=en_US&ref_=as_li_ss_tl
referrer-policy: unsafe-url
server: nginx
set-cookie: _bit=o540g2-c78698ce62c20457a9-00t; Domain=amzn.to; Expires=Sun, 01 Dec 2024 00:16:02 GMT
strict-transport-security: max-age=1209600
content-length: 395

HTTP/2 503 
content-type: text/html
date: Tue, 04 Jun 2024 00:16:02 GMT
server: Server
accept-ranges: bytes
x-amz-rid: 1NXFP2E2NDKH7SRW08J9
vary: Content-Type,Accept-Encoding,User-Agent
etag: "a6f-6187f291ddc80"
strict-transport-security: max-age=47474747; includeSubDomains; preload
last-modified: Wed, 15 May 2024 14:44:50 GMT
x-cache: Error from cloudfront
via: 1.1 646b6f21a2659c68f7a3822d035b97d2.cloudfront.net (CloudFront)
x-amz-cf-pop: NRT57-C2
alt-svc: h3=":443"; ma=86400
x-amz-cf-id: Ga9AudCUHmBDKRawpBRiIak02Bq7xlhXTqcZSsrJNS4ETzZArurkeg==
```

How to unblock certain User-Agent's on a Lightsail Bitnami Wordpress instance

Add your answer

Relevant content