Table of Contents
Host the API on its own hostname
Serve your API at
api.example.com, never at
As your API's usage grows, it will expand beyond your website/dashboard server
and need to move to a separate server or many separate servers.
You'll want to be able to move things around by just pointing its IP somewhere
else rather than trying to proxy things from your dashboard server.
Your API may also have more relaxed security restrictions in terms of TLS versions and ciphers accepted that you don't want to relax on your dashboard website that handles sensitive information. Having your API at its own hostname means it can have its own TLS certificate and TLS restrictions.
Also, when it comes to blocking bots and poorly-written clients, a user should still be able to reach your main website for support even if their IP is blocked from reaching your API server.
Don't be too liberal in what you accept
Accepting a slightly non-conforming API request today from someone's ESP8266 buried in a forest might mean you'll have to keep accepting those same non-conforming requests being made years later, often at the cost of having to implement workarounds in your web framework or server code as it gets upgraded.
A lot of users will hack something together until it works with your API and when it suddenly stops working months or years down the road, you're going to have to deal with the fallout. Rather than bending over backwards trying to support poorly written code, don't let their bad code function properly in the first place so it doesn't get deployed.
Though you don't have to be pedantic about it. Pushover's API has a message size limitation of 1,024 characters. If the message parameter is larger than that, I could reject the request because it's not correct, but then the user's message is lost and they may not have any error handling. In this case, I truncate the message to 1,024 characters and process it anyway (assuming it wasn't so large that it hit the web server's request message size limit). The user still receives something, and if they care that it's truncated, they can properly implement continuation or smarter truncation.
Avoid OAuth if you can
It's a confusing protocol that brings its own security problems and it introduces a lot of overhead for your users to get up and running. With OAuth your API can't be used from a simple curl request but has to be a custom multi-step process pulling in a whole OAuth library.
Use static API tokens if you can, but make it easy to rotate them.
If possible, avoid using an authentication mechanism that requires custom HTTP
headers (including basic auth) because some esoteric devices and plugins don't
Some don't even support HTTP
POST properly and will only be able to put form
parameters in the URL query string (though you still need to insist on requiring
POST method, don't be an animal).
Remember, not everyone is going to be sending HTTP requests from the ideal code you would have written, they're using what they have available.
Log a unique id with every request
This is probably done by your web framework, but if not, generate a unique ID or UUID with every request, return it to the user in the message body somewhere, log it, and ask for it on your support form.
This will make your life much easier in the future when you need to track down requests in log files and correlate them to a user's support request. Often times the user won't know the IP they're sending API requests from or their parameters aren't getting parsed properly so you can't search your logs for their API token.
Side note: you probably don't need a fancy centralized logging setup to capture
every detail of every HTTP request.
Filter out sensitive
POST parameters (Pushover redacts
one line per request including the
request's UUID to flat files, then rotate and compress the log daily and delete
old ones according to your retention policy.
In 99.9% of Pushover support requests where I needed to consult server logs,
having the date/time, sending IP, method, timing information, and sanitized
POST parameters logged was enough to resolve the issue for the user.
[2023-07-12 16:30:24] [62dd8009-3174-4b86-8078-18330a1e7b0e] [184.108.40.206] method=POST \
path=/1/messages.json format=json controller=Api::One::MessagesController \
action=create status=200 duration=66.14 view=0.1 db=14.73 \
"user"=>"uc65prcvfxxxxxxxxxxxxxxxxxxxxx", "device"=>"iphone", "sound"=>"carl", \
"message"=>"[FILTERED]", "monospace"=>"1"} cf_ray=7e5c6320196e0252-ORD \
time0=0.05 time1=3.15 time2=4.78 time3=4.81 timebuild=16.65 timesave=59.43 \
message_ids=m749182415778658654 queued=1 server=hippocampus
Be descriptive in your error responses
Assume a human will read them, even if it's unlikely. You might be surprised how error messages propagate but at least give the error a fighting chance to be seen.
When applicable, have your API errors include URLs to documentation to avoid support requests rather than just responding with a minimal "invalid xyz parameter" that the user has to decode.
Use prefixed tokens
This took me years to stumble upon, but use a short prefix for each type of
random ID you create.
Instead of generating an API token of
Mk7vuCg9eptiV8qid4mn, make it
Instead of a user key of
a for API tokens,
u for user keys,
g for group keys,
subscribed user keys, etc.
This makes it easier for users to keep multiple keys/tokens straight when they
all look like gibberish and it makes it possible to automate helpful API error
responses like "your token parameter has a user key instead of an API token".
Stay on top of failures
Reportedly half of all e-mail processed is spam. That means a lot of money, servers, and administrative overhead is wasted just scaling up resources to deal with junk that no one wants. If you don't stay on top of error responses from your API, they will accumulate and you'll end up wasting a lot of your own money constantly serving bad requests.
Obviously you'll want instant alerts of
5xx errors from your API when your
database falls over, but here I'm talking about the normal
4xx client errors
generated in response to missing required parameters, expired tokens, and other
conditions that you don't normally need to worry about.
Pushover's API generates about 1.5 million
4xx errors every day and each one
increments an expiring counter in its database for the sending IP (for IPv6,
rounded off to the
When that count reaches a certain limit in a short amount of time, the IP is
banned for one hour and subsequent requests short-circuit all of the API logic
and are responded to with a
429 status and a descriptive error message.
When an IP block can be mapped back to a user by way of an API token in the request, that user is sent an automated e-mail explaining why they were blocked and that they need to shut down whatever is continuing to send the failed requests.
If that error count keeps increasing past another threshold, the IP is blocked harder by adding it to a pf table, blocking it on the IP level. After 1 hour, the block is automatically removed. If the IP gets blocked again in a short amount of time, it is blocked for 2 hours, then 4, then 8, etc. However, avoid doing this type of IP-level block too early or you'll get inundated with generic "I can't connect" complaints. Let the requests fail on the HTTP level with proper error responses for as long as possible so they get seen.
Something I learned long ago is that automated things can have a hard time dealing with failure. Many people are also bad programmers and their method for dealing with "my HTTP query didn't get a 200 response" is to immediately hammer the API server again until it does (or it crashes, causing it to be restarted over and over). So a user's script sends Pushover normal API traffic all month until it gets to the final week, their API token reaches its monthly limit, and now the script is in a loop hitting the API with dozens of requests per second.
After dealing with this numerous times, my solution for Pushover's API was to
add a step after the soft block (where requests are dropped with a
status), but before the hard block (where all traffic from the IP is dropped).
After sending enough
429 responses that the IP's failure count is reaching the
hard-block limit, Pushover's API will temporarily respond to rejected requests
200 status code but with a descriptive message in the body explaining
why the message was rejected.
The user's messages weren't going to go through anyway because they were already
sending bad requests (their API token reached its usage limit or the input was
bad), so responding with a
200 shouldn't cause too many problems and the
descriptive error might possibly be seen by someone once they check why their
messages aren't being processed.
If these temporary
200 responses fail to slow down the client, they'll
eventually run into the hard block limit and get cut off, but it was worth a