How to avoid google to index cloudfront served content

0

Hello all, A customer whose cloudfront served pages are also being indexed and cached by google. So a google search for them provides 2 entries:

  1. https://www.domain.com/
  2. https://d12345.cloudfront.net/

Is there anyway to block the second cloudfront location from being cached? I've read on stackoverflow on using bots to avoid that: https://stackoverflow.com/questions/60123731/google-indexing-cloudfront-distribution AND Using DNS redirectionto avoid that: https://medium.com/tensult/how-to-do-site-redirection-using-aws-522a4002c645

Is there an easier way to do this?

Alok_T
asked 2 years ago395 views
1 Answer
0
Accepted Answer

Yes, there are few things to be done to avoid this from happening:

  1. Origin needs to be locked to the outside internet, except Cloudfront IPs. It appears that Google is able to reach the Origin directly.
  2. Validate if the *.cloudfront.net URLs are a part of the publishing code anywhere. If so, replace them with the Cloudfront - fronted domain names.
  3. Put a robots.txt on Amazon S3. Something like, the www.domain.com/robots.txt

I would not recommend putting Origin redirection etc. coz there is no reason for anyone to hit the Origin directly. It is also an open backdoor for the malicious actors to get through.

Once you do #01, #02, #03; the indexing of the Origin URLs will gradually go away. Also I believe, they can also reach out to Google (after doing 1, 2, 3) to remove those URLs from their search feeds.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions