Fighting API Abusers - Chaotic Good

One of the worst feelings you can have when running a website or service is to be randomly attacked by a bad actor. More often then not the attack will just be a Denial of service attack; which, while bad are actually pretty easy to deal with in the modern era with services like CloudFlare or AWS Shield.

An attack that can be more problematic is API abuse. This is where a malicious actor builds a website or service around your API without concent and begin funneling traffic your way for the purpose of fueling their product.

In this post we will cover:

• the chaotic good approach I took to help prevent a bad actor from using our Selfie2Anime service
• How to actually solve the problem properly with AWS WAF (Web Application Firewall)

Overview of the attack

On the 16th September 2020 I received a weird support ticket under my AWS account alerting me to the fact that a sizable portion of emails sent from my SES domain were being bounced.

"Your current bounce rate is 10.13%. We measured this rate over the last 10,109 eligible emails you sent. Our analysis covers the last 1.3 days."*

The ticket was flagged as Normal, so I wasn't too worried; however reading more deeply I went from mildly confused to very stressed.

"If we don't hear back from you or if the problem hasn't been resolved within the next 41,000 emails you send, we'll pause your ability to send email using Amazon SES in order to protect your reputation as a sender."

You heard it right, AWS support would have paused all SES messages outbound which would cripple Selfie2Anime entirely.

Investigating the attack in more detail I was able to see that the burst of bounced emails happened all at once.

I had recently setup X-Ray tracing on the stack so the first place I went for guidance was the service graph. I could immediately see that not only was I receiving a lot more traffic, but a fair few requests were failing.

Digging down into some of the statistics on what clients were hitting the service indicated a weird user agent of Amazon CloudFront.

This made it clear that the requests were coming from another host that was proxying requests and either proofing a user agent or something similar.

What attacks are there

Before diving into the details, it is important to understand what I mean when I say API abuse. The typical thing you will see people try to get away with if your API is discovered it to simply have the client connect to your API directly.

This approach is simple, and works really well if the API behind the scenes isn't implementing any form of Referrer-Policy checks.

Note: A Referrer-Policy check could be that the backend API checked to confirm that the website referring the user to the API matches a particular one allowed.

A more specialized, but equally simple attack could provide an API of their own however behind the scenes requests are just proxied to your API for fulfillment.

This particular attack is more annoying to deal with, as the request controlled on their server can forge things like referrer policies. Unfortunately this is the attack I was up against based on the information I had pulled from the user agent.

Stage 1 - Temporary Fix

Time was not on my side and every second passing was more bounced emails that were adding up on the total that would ban me from using the SES service.

Using the X-Ray traces from the first part of the investigation, I was able to find the email address domain that was causing us issues.

The domain of our attacker was post-shift.ru. Checking the website however it hit me that this was just a API driven temporary email address service. The issue was that they must have blocked my domain from sending too them and caused massive bounces.

I needed something done fast to stop the emails from flowing so I added some dirty code in the Lambda handler that receives the images and added a check to make sure that domain wasn't post-shift.ru.

email = body['email']
try:
    email_domain = email.split('@')[1].lower()
    if email_domain == "post-shift.ru":
        return response
except Exception as e:
    return response

I immediately saw the bounced emails begin to drop off and I was able to rest easy; or so I thought.

Stage 2 - Attacked Again

It wasn't for another 14 hours but I suddenly received a notification saying that there was a very high number of bounces coming through. Checking the metrics revealed that again, I was being attacked

I was quick to jump into action this time and found out that the requests were coming from a different temporary email site, this time 1secmail.com. I went to update the Lambda again with the new email but then got a magical ideal.

Return Fire

Whoever had decided to use my API service was likely harvesting email addresses of people and fronting it with their own Selfie2Anime clone.

The process would go as follows:

• User submits photo and email to fake site
• Fake backend receives the photo and email.
• Fake backend creates a temporary email
• Fake backend forwards the users photo to the real backend but using the temporary email
• Temporary email receives the users finished photo
• Fake backend sends this to the user under their name.

It seems like they've really got me caught! If I block the temporary email service they use they'll just find a new one. But they forgot one crucial thing. I am in control over what THEIR users see.

So I did what anyone in my situation would do... I began sending shirtless photos of older gentleman to the domains 1secmail.com and post-shift.ru.

The idea is that whatever users the fake site does get won't be around for long; cutting off the head so to speak.

How to properly fix

To fix this issue properly was actually very simple with the help of AWS WAF (Web Application Firewall). This service allows IP based rate limits to be put in place effectively slowing down services that might be piggy backing your API.

Below is an example of some CloudFormation that can be deployed and attached to your API Gateway resource to protect it from mass API calls from a single IP address

WAFv2WebACL:
    Type: AWS::WAFv2::WebACL
    Properties:
        Name: rate-limit-waf
        Description: Selfie2Anime Rate Limits
        DefaultAction:
            Allow: {}
        VisibilityConfig:
            SampledRequestsEnabled: true
            CloudWatchMetricsEnabled: true
            MetricName: rate-limits
        Scope: "REGIONAL"
        Rules:
            -
            Name: rate-limiter
            Priority: 0
            Action:
                Block: {}
            Statement:
                RateBasedStatement:
                    Limit: 100
                    AggregateKeyType: IP
            VisibilityConfig:
                SampledRequestsEnabled: true
                CloudWatchMetricsEnabled: true
                MetricName: rate-limiter

Summary - Did it work

It is difficult to say whether this solution is going to work long term, as the rate limiting is still able to be circumvented up to a point by deploying multiple services to multiple hosts in the cloud.

The solution where I send back lewd photos however will always be the fall back plan if the more conventional method isn't working out.

What did you think of both solutions? Which one did you prefer. I'm interested to hear if you have other ideas on how you would solve this problem! hit me up on Twitter @nathangloverAUS.