IPNs not being generated at all for live paypal.com AND sandbox

[Syrinscape]

2 days ago we started getting a large number of customer complaints from people who had subscribed to our service via PayPal subscribe button but we had not activated their subscription.

We checked our logs and paypal.com activity and ran some tests and confirmed that payment is being taken by PayPal, emails are sent from PayPal to users, but IPNs for MANY transactions are not be generated at all.

They are never sent to our server, and they never show up in the IPN history page. Searching the IPN history for a transaction that we just made where payment was taken, which does show up in the activity page, does not show up at all in the IPN history page, even 24 hours later.

We have a typical number of subscriptions naturally expiring every day, and in the last 24h that number has increased by over 400% as IPNs are also not being generated for recurring payment transactions, and our system has revoked access for some of these users based on non-payment.

We have an open case with Merchant Technical Support who have not responded in 32 hours, and PayPal Business support who say this probably won't be looked at until next week.

Can anyone confirm if a similar problem is happening, maybe undetected?

Is there any way to escalate to have MTS actually look at our case?

[Syrinscape]

Update: SOME transactions do generate IPNs and we do successfully receive and process those. But many transactions do not generate any IPN at all. So it is not a problem with our IPN handler or application or PayPal configuration, none of which were changed before this issue started happening.

[Syrinscape]

It is now day 14 and we are still seeing many IPNs not sent and missing from IPN history page every single day. We have still NO significant response from MTS on our case beyond "I have raised request to the product team to investigate this issue further" and "our product team are working on this issue". Beyond appalling.

[rachitskiy]

Hi, have the same problem and waiting for the resolution for quite a while as well 😞 

[Chednet]

We had a response from MTS today saying they believed the problem to be resolved, alas it has not.  We still have all of our subscription services disabled since this all began.

[Syrinscape]

It is now 20 days since the incident started and there are still many payments being collected every single day for which there is NO record in PayPal IPN history and no IPN sent to us.

We have sent MTS hundreds of transaction IDs which are missing IPNs to investigate. I believe their response indicate that they at least confirmed some of them were not appearing in their logs, but it was pretty vague and I am not sure I correctly understood them. Still no ETA at all on when someone will even look at the issue.

In the meantime we have resorted to a daily manual export from the "Activity" page and implemented an import feature that creates a faux IPN from that exported "Activity" page data that our system can process.

Running this import with an "Activity" page export from the past 6 months reveals that we have typically about 0.09% of transactions for which we never receive an IPN. Since this incident started, that number has risen to 3.9%.