IPN notifications seem to now be triggering HTTP 406 responses from Apache

[QWeb_Ric]

We've built countless online stores now in a bespoke framework which uses IPN notifications to watch for payment completion etc. All have been working just fine for years, and all are hosted on CentOS 7 or 8 servers, running Plesk with near identical set-ups.

We pushed a new website live yesterday and the client reported that Paypal payments aren't going through. Digging, we found that the IPN notifications are stuck in a "retrying" state and the server is returning HTTP 406 responses to these requests, "AH00690: no acceptable variant".

Digging further back, it looks like the last successfully handled IPN on this particular server was a few days ago on the 16th. The issue is affecting multiple websites but so far potentially only on this one server despite the handler code itself being identical across all of our websites on all of our servers.

Googling I'm not finding anything written about this over the last couple of weeks, but reading between the lines of various older posts relating to this particular error code and how Apache processes requests internally, it's sounding to me like the IPNs have started to be pushed with Accept headers which Apache on this particular server doesn't seem to understand.

Plesk did pull some updates recently but their changelog doesn't seem to reference anything of relevance. I also see from the Paypal status pages that maintenance work was actioned on the 15th and that a couple of incidents seem to have been reported and were being worked on yesterday, but again not really anything that sounds relevant.

The IPN url we use is /foobar/paypal-ipn without any suffix. We use Apache Multiviews to map that up to /foobar/paypal-ipn.php and the above error suggests that it's Multiviews not knowing to server a .php file because of the Accept headers used by the notifications, but I've already tried to resolve by by setting MultiviewsMatch Any, and by using a manual RewriteRule instead of Multiviews. Neither approach makes any difference.

Surely we're not alone in experiencing this? Any advice would be appreciated.

[cambo]

We are seeing this at my company too.   We started to see intermittent 406 errors on Oct 1, but unfortunately our monitoring didn't pick it up. Then on Oct 16, 100% of IPN callbacks were failing with that HTTP 406 error.

It is still affecting us right now.

We have submitted a Merchant Technical Support ticket and have been told their Engineerings are working on it.

This is the third time in the last month that PayPal has released bugs that have resulted in full or partial outages of our PayPal processing systems.

[QWeb_Ric]

Ah, a relief to see it's not just us then. I'm surprised there aren't tonnes of posts about it given the number of people who must have a similar setup. Strange too that it's been happening for you guys since October... Did you pull any Apache etc updates at around that time?

[cambo]

I don't think we made any Apache server changes at that time , no.  

The pattern is very representative of PayPal.   They release some new changes, but it takes a while to propagate completely through their servers.  So there is intermittent issues for a period of time, until they route all traffic through the servers with the new changes. Then it because 100% failure. We've seen this multiple times from PayPal over the years.  

[QWeb_Ric]

Yeah that makes sense, thanks. I guess we're just going to have to wait this one out then 😫

[QWeb_Ric · Accepted solution]

Ah, I've managed to sort a work-around at last. @cambo I hope this works for you guys too?

The issue appears to be entirely with Multiviews unable to understand what Paypal is now asking for. As per my original post, presumably the notifications are now coming through with Accept headers that Multiviews can't work with.

Having already tried various iterations of adjusted .htaccess configurations in the website root, where Multiviews is enabled, I've given up and created a new .htaccess within the folder that our script resides. I.e. in my original post I said that our callback url is /foobar/paypal-ipn (it's not actually foobar obviously - just masking here to avoid any security implications!), so I've now created /foobar/.htaccess to capture these. For me, the following .htaccess config seems to be working:

RewriteEngine On
RewriteBase /foobar/
Options -Multiviews
RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^(.*)$ $1.php [L]

So for these requests I'm now totally disabling Multiviews and using a basic RewriteRule to trigger the same routing. You'd need to adjust if you have other requests going to this location which need need routing to say, mp4 files, because mine will only look for and route to .php files to serve such requests, but presumably you wouldn't have anything other than processing scripts in this location anyway.

Hope this is of use to somebody.