Editing Robt.txt for Cloudways CDN


#1

I found that using the cdn from Cloudways has an issue. The url isnt whitelisted with twitter, so images do not show on twitter. This is a common issue with all cdns. The solution is to edit the robots txt for your cdn to ALLOW the twitter bot. I want to know how to do that. Anyone have solutions here?


#2

Hi @jbaer,

Currently the robots.txt is default for CloudwaysCDN. Can you let me know what rules should be in the robots.txt to make it working with twitter?

I think you can also set TWITTER CARDS to make images, titles and descriptions appear on Tweets as well.


#3

This matter is new to me and I am just learning about it, so please forgive any misunderstandings. Ill explain the best I can to help get this resolved.

One can add meta data to their site to activate the twitter card. I did that long ago and my posts did show as expected on Twitter.

After adding the cdn, my meta changed from
`
to

`

After researching, I found via Twitter documentation, that the image does not show in my posts because twitter bot can not crawl my cdn. So I attempted to add that to my robot txt to see if that works, but I suspect it needs to be done at the cdn, not at my site. This fact is something I dont know for sure.

A problem with testing this is that twitter may be caching, so even if I fix it, the fix may not show for hours or even weeks. Source: https://twittercommunity.com/t/twitter-caching-your-robots-txt-file/15314/17

Long story short, I am experimenting with my robot txt, but I think the issue is that the cdn is blocking twitterbot. Ill keep this thread updated as I learn more.


#4

After trying to get his to work for almost 10 days, I exchanged Breeze cache plugin with W3 Total. The problem was instantly fixed. Im not sure what occurred, but breeze was preventing twitter bot from accessing my images, which prevented them from showing on twitter.


#5

Hello,

Just found this thread while searching solutions for the same issue. Is there any way to fix the issue without replacing Breeze cache plugin? Please advise.