502 bad gateway errors


#1

The past few days I have seen a huge increase in 502 bad gateway errors across all sites and servers. Mostly WordPress sites but not all.

I contacted live chat support and they said this issue has been reported by lots of customers recently and they are investigating. I have a ticket open but they suggested it was caused by a plugin I had activated on one site, they are investigating further as this seems like a wider issue.

It’s not an issue with my servers running out of resources.
I am running Digital Ocean servers.
The 502 bad gateway errors seem to last for about 5 seconds, after which refreshing the page works.

I’m keen to see who else is having this issue on their sites and if there is a pattern.


#2

I too have the same issue, they told me it was due to the ManageWP worker plugin. Do you use this plugin? Problem was really bad a few days ago, seems better now and I haven’t changed anything.


#3

Yes that was the plugin I was using on one of my sites and they told me to disable. However I have a second server where no sites are using that plugin, also seeing lots of 502 errors.

I have got an update to my ticket and they are now saying it is related to an attack where they are trying to access the dump file/.sql files. They have configured something to block these attacks so we’ll see if that works.


#4

We started to experience this on ~Monday on our primary server hosting WordPress clients. We thought the issue was limited to that. We were told by support to block access to xmlrpc for all sites and follow the best practice of disguising the wp-admin area. Today we implemented those changes, but then started noticing intermittent 5-15 second 502 Bad Gateway errors on our secondary server as well, which hardly hosts any websites and gets very little activity.

Both of these instances are with Linode.

If a plugin is in fact the culprit, luke1 - perhaps we should compare our list of commonly used plugins?


#5

I highly doubt it’s plugin related. But the good news, on my secondary WordPress client server, there are hardly any plugins used. Akismet and Vevida Optimizer are two that could make outside connections on their own with a timed procedure.

I disabled Vevida first and after a few minutes, 502 Bad Gateway.

I now have Akismet disabled from all sites on the Secondary server and so far no 502 Bad Gateway. Still experiencing them intermittently on primary server.


#6

I’m feeling very confident this is related to connections to Akismet. Testing this will be difficult though on our Primary server but I’ll do my best.


#7

Can you check your nginx-app.status.log file and see if there are any entries such as:
502 42.200.106.20 [24/Sep/2019:07:44:10 +0000] GET //backup.sql

Search for 502 and it will find the 502 errors, then does it mention a GET request relating to .sql


#8

It’s not Akismet. Looking into your log question now.


#9

Where do I find nginx-app.status.log?


#10

Login via SFTP, go to /logs in the top folder and sort by last modified to get the most recent version, or find the log file from a date where 502 errors were occuring.


#11

My environment does not follow this format. There is no /logs/ on the top folder. When I SSH in, i can go to /var/log but I can’t view any of the .log files due to permissions issues.


#12

@luke1 - Problem has been solved.

We were receiving thousands and thousands of random requests to download .sql files from a massive botnet.

CloudWays support added the necessary config code to nginx to block GET requests on .sql files. So far so good… the longest amount of time with no 502 Bad Gateways yet.

I’m relieved this didn’t have to do with CloudWays’ architecture or plugins I’m using.


#13

Good to hear, that sounds like the same issue as mine. Modified nginx and no more 502 errors.


#14

I am getting this error now on multiple servers, VULTR. Just started in the last week or 2.

Whats going on?


#15

You’ll want to contact support and ask them to check your server logs to see if there is an attack targeting your .sql files. They can modify nginx to block it.


#16

I still have this issue (and only since roughly a week) also for custom-developed services and APIs, completely unrelated to Wordpress or other CMS.

As advice about Bad Gateway remedies talks about possible server overload I secured that there was plenty of room left on the server (there was maybe 5GB left, which should have been sufficient anyway).

I also checked use of the server, but didn’t find anything different from now and months ago in terms of increased traffic, RAM use or so. Actually, no parameter had any jump at all.

I use DigitalOcean, and someone else mentioned Vultr, so my closest guess would be an issue with Cloudways’ DNS, as other accesses would go directly to each hosting provider. I don’t use Cloudways’ CDN etc.

Anywho, I will check for accesses to backups and .sql files.


#17

Just realized DigitalOcean’s DNS is used, so that’s most likely not the reason.


#18

I’m seeing this too on a couple of sites. Both are on Linode servers. I can see from the logs that it is definitely related to illegitimate requests for .sql files, and each one causes a 15-20 second outage with a 502 error. I’m not expert enough to know what mechanism could be bringing the server down for a short while just by requesting a non-existent file though.

It looks like the fix needs to be applied per Cloudways user or site then. I’ll put in a ticket now…


#19

Hi all,

An update about the 502 errors. Our support teams have investigated the issue and it turns out to be DDoS attacks that hits your sites on 404 pages, the attacks keep increasing and they eventually crash your server.

Please contact our support and they will apply the fix on your websites, either by blocking the requests or pool of attacking IP addresses.

  • Ahsan

#20

My applications are under /home/[server-alias]