Digital Ruby Software – Home
Greetings. I’m Jeff Johnson and I founded Digital Ruby in 2009 in order to learn how to run a software company, teach myself mobile programming and expand my skills and knowledge. I craft all manner of software including mobile apps, websites, security and even Unity and visual fx.
As some of you know, ipban pro had an outage on 2022-08-04. This lasted a good 6 hours.
I have been using Azure Functions as a proxy layer for all of ipban pro. At a basic level, Azure functions knows about all the other services and data stores necessary to handle orders, licenses, api keys and the recent and naughty list. For the last several years it has been running pretty smoothly.
Yesterday I was ddos attacked for about an hour. After this attack, Azure functions decided to melt down and return 503 codes or timeout for all calls. I believe this was a bug in Azure functions not releasing connections. My metrics showed 600 simultaneous connections and then suddenly zero every few minutes. Clearly Azure didn’t like that and decided to kill my app. Even stopping it for a few minutes and starting it again failed to fix things.
I had considered just deleting the function app and re-deploying it, but in my mind the root problem needed to be discovered and addressed.
I spent a few hours poking around application insights and the troubleshooting section. There were references to this connection limit. But I even shut down my entire layer of services and still received the 503 and/or timeouts when calling the function app directly. So it wasn’t because of traffic I was throwing at it.
Azure charged me 29$ to just be able to send an email to someone. By the time they responded four hours later, I had re-written my function app into a regular server app and hosted it myself. Azure still hasn’t provided a reason as to why the function app went belly up.
This is the first time I have had a cloud provider completely fail on me. I won’t be using Azure functions again. This also left a very sour taste in my mouth for Azure itself. The other services Azure provides have been working well for me but I will definitely be keeping a backup plan in my back pocket in case some of the other services have similar problems.
Storing terabytes of data in Azure, AWS or Google Cloud can get quite expensive. Per Backblaze, here is a brief breakdown of pricing as of June 2022:
Backblaze does not charge for incoming bandwidth, upload or delete requests.
For the big 3 in cloud storage, 10tb of data will run you between 2000$ and 2500$ per year – just for the storage. This doesn’t cover the api calls or bandwidth which can often be an order of magnitude greater or more than the storage cost – running your bill to 25000$ or more.
A couple of new providers have really taken off that are not in the above table.
Cloudflare is unique in all storage offerings in that they will automatically replicate your data for you in multiple regions. They attempt to move objects closer to where they are accessed. This makes Cloudflare R2 a top choice for any data that must be highly performant and redundant.
5.99$/TB/month. Wasabi has a sneaky provision. Each file will round up to a minimum of 4k (but not higher) when determining billing. So even a 1 byte file will count as 4k for billing purposes. Also, deleted files will count against your storage for 90 days after deletion. But they have no API fees and no bandwidth fees.
Wasabi has a clause in the contract that if you go over bandwidth in a month that is greater than the sum storage of your account that they might terminate or suspend the account without notice. Yikes…
There is a common trend of vastly reducing bandwidth costs or not charging for bandwidth at all. This will force AWS, Azure and Google Cloud to adapt, and I could see a future where they all move to an api call + storage only billing model (or even storage only). They will have to adapt or start losing market share as tech companies go to the cheaper providers.
The last point I will mention is bandwidth lock-in. This is why the major cloud providers charge for bandwidth out. For a large businesses it becomes prohibitively costly to migrate their data out to another provider. Essentially a form of blackmail to ensure people don’t switch to another provider. It’s wrong and is why I respect Backblaze and Cloudflare so much, along with other companies who are part of the bandwidth alliance.
It was about 7 1/2 years ago I quit my last day job at Ancestry.com. Now today I have left Jane.com as the vp of engineering due to personal health reasons and misalignment with top company management. I am taking a breather for a while and then plan to go full steam on IPBan Pro to ensure that it solves the problem of hackers, botnets and brute force login attacks.
To achieve this lofty goal I have to give it every ounce of time that I have. I still battle my brain issues but am becoming tougher and able to deal with them a little better, along with learning the diet and medications that help things.
Here’s to another great adventure!