This is painful, I was building for fly.io because of their dynamic request routing but I'm thinking I'm just going to go with hetzner. The cost difference is so substantial. If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
We're sorry. I hope we're clearly expressing what's happening here: either we do flat pricing globally, so that people deploying in cheaper regions subsidize those deploying in more expensive ones, or we reflect our cost basis in our prices and let our customers (and prospective customers, like you) make their own choices.
In your specific situation: it really just is the case that Brazil soaks us on import fees. Getting machines in racks in Brazil is just crazy expensive. We're going to keep doing it! Brazil has too many good sandwiches. But the idea behind building a new public cloud on our own hardware is for us to be around for the long haul, and flat low global pricing is not a "long haul" decision.
(It wasn't a long haul decision last year, either, but building a billing system capable of expressing this stuff is a nightmare from which we have not yet fully awoken. It turned out to be a shockingly hard problem.)
There are going to be cases where places like Hetzner make a lot sense compared to us. We hope you get your thing launched and find success wherever you end up. Some of our stuff, like LiteFS, you can take with you! :)
I think the change makes sense, even though I have workloads running on the regions that got the steepest price changes. I'd rather have more granular pricing than a blanket increase. The why now also makes sense - there was a technical limitation that doesn't exist anymore.
I was expecting a larger grace period from announcement to price change though.
You got it right when you started charging for stopped volumes - you overcommunicated and gave users estimates for a few months before you started charging.
Smoothing the increase over the next 4 months was nice.
A couple of nits as well:
- AFAIK prices increased in all but two regions. The announcement could be clearer or more direct regarding this.
- The pricing page could have a clearer from/to. I have to click on the example toggle to know how much machines cost before the change.
This same argument applies to most cloud hosting, non-cloud (or less cloudy, Hetzner is a little cloudy) will always be cheaper because the service is more basic in a number of ways.
Fly’s pitch is not just on demand scalable compute, but also that compute being close to the edge, and with “ops-less” deployments. All of those are factors you pay for.
You could pay a little less with a big cloud provider’s serverless platform, a bit less again with regular cloud VMs, a lot less with colo hardware, and a ton less running your own DC (at scale). Obviously not all of these work for all businesses, but saying Hetzner is cheaper than Fly misses the point of those two services on the spectrum of service levels.
This has me so confused. Should the cloud abstractions not clearly make the service more optimizable at scale and thus cheaper to offer? Sandboxing things, limiting options, not having to deal with users wanting to provision VMs and do all kinds of unexpected stuff? The marginal cost of the beautiful software to make all of that happen smoothly, and all the dx it affords, that will just go towards 0, no?
I understand that people ask for what they can get away with. But does the market fail or am I missing a piece of the cost/price puzzle?
- Things change and break. You are paying for people to fix that for you at 2 AM, and to keep up with industry developments and best practices and make those decisions for you.
- Legal risk and compliance requirements change. You are paying for them to keep up with those for you.
- Overhead imposed by multi-tenancy. Security boundaries, control planes, etc. These are fixed operational costs caused by the abstraction.
You're not paying for just an abstraction, you're paying for the things the abstraction enables. It's also not always what you need. But it often makes good business sense.
At one side, the big cloud providers have their very fair profit margins. But I've always assumed that comes mostly from that they can, and do, charge extra for the beautiful software (not seldomly through ridiculous egress bandwidth pricing).
I'm skeptical of how much flexibility and optimisation the big cloud providers really get compared to a very large VPS provider?
All the compute products (different form of container hosting, serverless) are essentially still priced at CPU+RAM + a premium. Some additional abstraction for nouveau databases ("capacity" or "billing" units). Many of the highly abstracted offerings also have the scale-to-zero concept, while the provider still have to have physical capacity ready (to some extent).
Sure, you can probably crank out some percentage more utilisation, but that doesn't really matter for the 100+% price-add on the market bears.
But I of course have no data to back up my claims, just speculation.
On a systems engineering level, keeping a system in a state of low entropy for a prolonged period of time will always be harder than a state of higher entropy. In practice, this comes down to more complexity in the software involved, and the people required to make the machine go boop will inevitably higher, since you need to maintain the servers and the complicated stack on them.
The tooling for managing the "lower" layers is progressively worse / slower / harder the closer to the metal you get,
so you usually are paying a heavy premium for convenience / DX.
Only very specific serverless products where the provider can bin-pack you with other tenants allow them to possibly offer lower prices.
The sandboxing (by way of example) is a hardware affordance, not a software feature. In order to hardware virtualize every instance running on Fly.io, we have to provision hardware globally; we can't run customer jobs in namespaced userland containers under a shared kernel on cloud-provisioned hosts.
IMO it’s rather sad that virtualization with native-like performance isn’t available on ordinary cloud instances. Getting nested virt or some kind of hypervisor-assisted partitioning working is hard but far from insurmountable.
For a lot of people (most...?), Hetzner gives them exactly what they're looking for at an excellent price point. As an ecosystem, we never really moved past VMs, we just went up the stack more.
I'm not missing the point of the service, I'm against the idea of infrastructure being an 84% margin business. But I guess this is what the VCs demand and some are willing to pay it.
Don’t host anything important on Hetzner, as they’re cheap they get a lot of questionable users, and their systems are tuned to null route anything that looks remotely suspicious or high-volume to their hair-trigger network monitoring and DDoS protection.
And then you’re at the mercy of whatever support agent decides to look at your ticket and that can take days.
I disagree, at my company (elest.io) we have several thousands VMs with Hetzner and reliability is very comparable to AWS from our internal stats.
In case of DDOS attack we get notified and can discuss with their team.
Also support is competent and quick. We usually get an answer in few hours most of the time.
I’m glad you’ve had a positive experience, maybe it’s the usage level that grants you that treatment. As a one-time new client I had the worst possible experience with them: null routed my game server on launch day, very slow to respond support team, and effectively held my server hostage for 3 days. I guess I needed to inform them I was intending to use the server I paid for?
there is no way Hetzner is worse than fly for uptime. Fly has deleted my db, taken servers offline for hours, and expired tls certs for hours without telling me.
Absolutely not defending fly.io. OP is migrating off fly and in my personal experience, Hetzner is not a good option. Maybe OVH, AWS, GCP, Azure, Render, Digital Ocean, or linode.
How is it you can have a good experience, and yet my negative experience somehow doesn't qualify and you're insinuating the issue is with me, and not Hetzner?
This happens with every provider. One thing you're seeing is just that people with 5- and 1- star reviews have a lot of motivation to post reviews. People tend not to be motivated by "meh it's fine" feels.
> If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
Please give us an update here when you are done building your hetzner beowulf cluster and let us know how it fares. I think you are grossly underestimating the system you describe.
Whether this is completely out of the box or a multi-year engineering project to get equivalent levels of utility in the long term is completely up to how one specifically uses Fly.io.
E.g. a load balancer in front of some relatively stateless http microservices is really not going to tie you to Fly.io because of complexity. A globally distributed edge compute environment with database, RPC, varying scaling patterns, and environment certifications is probably not going to have the value prop impacted by this kind of minor pricing change though.
True, but managing 5 servers requires overhead. Where does the load balancer sit? What happens when it goes down? What if the entire datacenter loses comms? That is why cloud providers have AZ's that are physically separated from one another at different sites. When your boxes are spread across different physical networks, how do you connect them securely? For true HA you need a system that can self heal. Just throwing servers at the problem does not make things better. Reminds me of the expression 9 women can't make a baby in 1 month.
None of this is impossible, but when you are a small startup it makes sense to offload this to someone else and not waste time reinventing the wheel.
For the record, hetzner cloud also has multiple data centers (at least in Europe, haven't checked for US) with sharedand load balancers . What you don't get is managed services like k8s, databases, sqs, lambda/faas. These are generally easy to setup via preconfigured Terraform or ansible scripts, but if something goes wrong, you'll have to look at the logs yourself - that's true. (But that's ignoring that such a small team as you're proposing would likely be better served with a more mundane setup, avoiding all of these extra points of failure you'll have to manage... Making a simple bare metal setup once again the less involved and better performing alternative)
Im pretty sure you're vastly underestimating the amount of performance you get from today's hardware.
You won't be a small team by the time you're at a scale to need horizontal scaling. Only the biggest consumer facing/social media sites get value from that.
If you cannot tolerate downtime you need a redundant load balancer and you need nodes in different datacenters. Scale really doesn't matter in this equation. Now maybe you don't need five-nine's so then the risk is worth keeping it simple.
I read it. Historically Hetzner has been popular in the dedicated server space. I cannot speak to their IAAS/cloud offerings, I have never used them. I will take them for a spin, though, seeing as it is really the only way to use their services in the US with good latency.
Hi, professional sysadmin here. I've run production deployments across hundreds of machines in Hetzner; it scales fine. The only real problem is that Hetzner is so much cheaper than anything else that it's easy to get a little bit stuck. (Honestly, that was a fun job in a lot of ways. Oh well.)
Never said it couldn't be done. I forgot that they have ventured into the iaas space and are doing more than just dedicated boxes now. For a small startup with a simple app, managing five dedicated nodes and building your own orchestration on top of deployments, load balancing, networking etc feels like the juice ain't worth the squeeze. For their cloud offering, it is more compelling.
I get that this stings, but honestly, outside of the brief outages when I'm trying to deploy for the millionth time, it's been a dream to run on Fly, with instances close to my users.
It's too bad the closest region to Dubai is Romania, and Africa/SE Asia are a bit uncovered, but that's ok for now. I'm really enjoying using LiteFS and globally distributed instances for each customer, making it really easy, fast, and scalable. As the sole dev on the team right now, this is fantastic.
I wish Upstash Kafka was a bit more baked, but I ended up just signing up with them directly. I had trouble using the built-in-to-Fly version, but that's fine.
There's also physical limits though, and structural economic lag in terms of investment. If you look at any CDN costs globally and compare it with the number of intercontinental fiber connections between those places you generally start having a good idea that the two are related. In some places there's also just less qualified talent (due to having less ability to train people in enough universities, or just not having them for as long) and datacenter space built and extra import costs that delay everything.
At some point everything will saturate and probably energy production costs will dominate (largely for cooling), but I think we're still a bit far from that.
Do you have numbers to support it? If you mean import of hardware that do makes sense, but what about hardware that's already here? (I'm also Brazilian)
Looking at AWS site it explains that PIS/COFINS and ISS are charged, excerpt below:
"AWS SBL can issue an NFS-e to a Bra zilian customer only after receiving a valid Tax Registration Number ("TRN") (i.e., CPF/CNPJ). The relevant taxes applicable to AWS cloud services offered locally are the following: PIS and COFINS (9.25%, combined rate) and ISS (2.9%, charged by São Paulo municipality)."
> but what about hardware that's already here? (I'm also Brazilian)
It's ridiculously expensive.
Very little hardware is produced domestically, and the little that is, you don't want to be running your switches on what is in essence a no-name "intelbras" brand.
I won't deny that I'd love the challenge. However, there are some caveats:
1. AINAL, so I might say legal matters wrong for not understanding completely how the tax law works in Brazil, (Tho I do like to search about it), but I might find someone who could help you guys with it.
2. Currently, there is a law already approved (now being regulated with complementary acts) about how taxes will change in Brazil to the VAT model, this could increase/decrease taxes for a service like yours.
Besides these 2 points, we could talk via email (xunjin.coder@gmail.com) to might find a way like AWS did, creating a Brazilian subsidiary, to be able to generate NF-e in accordance for each valid Brazilian Tax Registration Number (CPF/CNPJ) and issue the invoices in BRL.
Seriously, thank you for offering. I'll let Scott and Dave know, and one of us will get back to you. (If anything comes of this, and you're OK with it, we'll write it up so people on HN can read about it; I always feel a little squicky "moving" discussions from HN to private channels).
Be my guest in writing about it, I hope I can help you with it.
PS: Life is sometimes a funny coincidence. My brother, who is a State attorney where I live, called me to share private matters 20 minutes ago, and already asked him for some help/directions about it. This weekend, I will speak to him in our spare time about it.
Great!, then I'd rather my app be a littler slower for those people if it saves me money. If the latency is not noticeable, why pay more to have that edge hosted.
Recently evaluated Fly.io for a company project but decided against it. I get the vibe that it fits the hobby/side project market more and that they're still in startup mode.
The reason to like fly is you get a 20-ish line yaml file that replaces (and I'm not exaggerating) thousands of lines of cloudformation. It automagically sets up the equivalent of a load balancer, autoscaling group, VPCs, db, vpn, etc. And it's trivial to replicate that environment per employee.
Plus the autoscaling using their firecracker stuff is fast.
I'd recommend most companies who aren't using tons of aws services take a hard look at fly.
I can’t find it in their docs but they stated it in the announcement email for pay-as-you-go.
I also have received a email saying they covered my ~$0.18 bill for the month.
You could probably contact support for more clarification.
The nice thing about Fly is it’s very docker oriented, so depending on your app migrating it off may be fairly straightforward?
My apps are all on Fly but in the regions without big hikes thankfully, so I’m sticking around for now, but it sucks some people are being hit by this…
Technically you are right. However my app is using litefs. In theory you could set this up on your own servers. Which is part of the reason I was so interested in it.
However with fly it’s just one command to add a new region and my data is auto replicated there.
That’s super nice compared to manually managing this.
As long as bandwidth costs do not go up drastically I think it’ll be okay.
Fly is the only place I can get a lambda-like application with an attached SSD. Tigris is cool too (s3-alike but faster for our use case).
FWIW, I haven't had too many reliability concerns but I have had some issues with builder VMs (can always build locally so no big deal) or non-prod environments with just one instance.
> If you were on the free Hobby plan at the time that the paid Hobby plan became the default for new organizations, your plan is now called the Legacy Hobby plan. Your costs stay the same as they were, with no monthly subscription fee, and no included usage beyond the free resource allowances.
so i get to use the legacy hobby plan because i'm grandfathered in, nice. only use it for a hobby project with one instance anyways(see bio), if they really decide to start charging me i'll gladly leave.
Heya, we deprecated pretty loudly! We made a post on the community forum [0] and we sent an email to all active accounts saying the following:
> The first improvement we're excited to announce is that the $5 Hobby plan is no more. We're replacing it with a very simple Pay As You Go plan. On this plan there's no more upfront $5 charge and no minimum monthly commitment. You only pay for what you use. If you don't use anything for a given month you pay $0. You still need a credit card on file to prevent abuse. But your card is only charged if you use the service.
If anyone's ever wondering how carefully choreographed our HN presence is, please refer them directly to this subthread, which, for posterity's sake, I will note is occurring on the morning of July 4.
Sorry, didn’t intend for throwaway to come across as not standing behind the product.
I wrote the community forum announcement and was one of the people that worked on implementing our PAYG plan. I just wanted to clarify that we didn’t deprecate silently, so I quickly created an account to do so.
random side note, it would be nice if the region selector for regional pricing would display human readable region labels in addition to 3 letter acronyms.
Whoa, they did get rid of it. Not surprising in the least, but they definitely did it with a minimum of fanfare. When Heroku got rid of their free plans at least they were upfront and released a blog post about it. Maybe they did and I just didn't notice.
We looked at both and chose Render based on various posts here along with our real world testing and bake off parameters. We use the Pro Ultra config and handle ~20rps on a Node / Express stack w/ 36 workers using clustering. Our bandwidth is around 50Gb / month.
Render has been quite solid and the support has been on point when we have found issues or run into unexpected edge cases. I have been impressed that despite a big raise and subsequent scaling up, they continue to ship a solid product and the platform improvements have been useful as well.
Never used Fly.io but I have a website running on Render (2.4M unique users per month) and the service is getting quite nice. Occasional hiccups, but they are improving. Not as stable as Heroku used to be 7-8 years ago yet, but it's also waaaaay cheaper.
I really like their Docker support and infra as code, makes it very easy to spin up a whole thing while not being too far conceptually from Kubernetes for example.
I have services running on each. Fly's more flexible. Render's further from the metal, and thus somewhat more limited.
Both have some reliability issues and rough edges. My new stuff is mostly on Fly, but I'd use Render where I had a junior team who had zero infra chops.
Very happy with Fly.io. My website Pillser runs on it and I never had issues with them. The cost per month us just under USD 100 for hosting 3 services, including the database.
Hetzner gives you bare (or virtual) metal that you have to install stuff on, and maintain it.
Fly.io and similar give you a higher abstraction layer where you throw a Docker container and the stuff below and around it is managed for you. Basic suff like OS patches, but also autoscaling, load balancing, request routing, etc.
Does the 4x difference make it up? It will depend entirely on you and how good you are with the things fly.io abstract for you, and how much you value your time spent on it.
If you are literally just looking for a place to cost-effectively park a VM, you should very seriously consider Hetzner! It would be weird if there weren't use cases where Hetzner comes out ahead.
Don't know if anyone noticed, but Ben Johnson recently became head of product over there. I'm fortunate to have called Ben a friend for a long ass time now and I gotta say, I'm extremely bullish on fly as a result of this. Ben is not just a developers developer, he's genius level intelligence. https://github.com/benbjohnson
(We fumbled the ball hard at DigitalOcean ngl, fingers crossed Ben can build what I dreamed of building one day!)
I’ll be bullish on fly when the awful outages stop. They had a hardware outage in the LHR region for a month, it was probably longer but I moved to a different provider so I haven’t bothered to check.
How has Flys reliability been as of late? From past discussions I got the impression that despite it being quite polished on the surface, behind the scenes their operations are a bit of a mess.
From personal experience, still not great. Sometimes things just don't deploy.
I host my little mobile braai simulator on it (braaisim.com) because it's an art project and easy to add new domains / certificates. Convenient to be able to scale up easily if it went viral in ZA. Nice that they support Johannesburg region (jnb), though, which is close to me.
You were downvoted, but as a user, I’ve experienced the same inconsistency with deploys.
There’s also a hard container size limit I’ve run into multiple times. If you add a dependency and go over the size limit, your app won’t deploy unless you switch to a GPU instance, which is substantially more expensive.
Don't switch to GPU instances just to get around the container size limit!
What are you trying to do and what's the size of your container? My instinctive response here is that you're not holding it the way we expect you to: that some of what you're sticking in your container, we'd expect you to keep in a volume.
I see two sibling comments with reports of patchy reliability, so I'll note that it's been rock-solid for me (multiple daily deploys every weekday) since I last commented on it (about a year ago?)
A bunch of mini outages and the main site + dashboard was out for maybe an hour last week on Thursday.
my production app has had 1 outage where stripe webhooks weren’t delivering for 45 minutes which is a huge issue for sure.
I mitigated by checking in addition to the webhook event and also it hasn’t had a problem since. Their explanation was a 3rd party networking issue at or near one of their data centers
Bad enough that we had to migrate our key customer-facing platform off of it because of the impact.
I love the idea and will still use it for many less-business-critical apps but after multiple late-night weird un-debuggable database outages we had to switch.
To provide at least one positive example, I'm using fly.io to host a website I made for a friend (for her college capstone project), so I'm on the now-legacy hobby plan paying nothing, and I have had zero problems.
I use it to power CI builds (a lot of them!) and have extremely little issue with it.
Basically I’m just using the API to spin up machines, which do some work and shut down. There’s some extra machines per build job, like database containers or a headless browser for testing. Pretty smooth in my experience.
I think the only occasional issue I hit is the internal DNS being a few seconds behind reality.
Semantics aside, if an incident/outage/<other term> affects devs/users in any way, it really should be on the status page.
I found it impossible to distinguish between user error and platform outage. Too often it was a problem on fly's end yet the status page gave nothing (perplexed, I'd rerun the deploy a few hours later and it would work).
Can't stress it enough: if fly's services aren't working for any reason, big or small, put it on the status page. Devs need to know when something's not in their control so they can inform their team and customers and go to bed or take a coffee break. They shouldn't be up till 5am thinking they borked something when the problem is 100% on fly's end.
We have the same semantics. The incidents that are on the infra-log and not on the status page are things that didn't affect devs/users in any way. A good example: we have clusters of edge servers in every region we serve. They're servers, part of their job is to occasionally throw a rod. When that happens, we pull them out of service and stop advertising them. That's an internal incident, but it's not a status page incident. Preempting a response here: no, I don't think users would be better off if we status paged stuff like this.
Every week, we go through the incident management system, and all our incidents, whether or not they had user impact, get written up here.
One thing you're probably running in to, which is not on you but rather on us, that we are aware of, and that we are grinding steadily away at: as the platform has stabilized over the last year, an increasing share of problems users run into aren't "platform" issues (in the sense of: things going wrong in our API or on our cloud servers) but rather bugs in `flyctl`, our CLI.
We have two big reliability projects happening:
* We've finally landed our "white whale" of being able to move workloads that have attached volumes between different physicals. I'm writing a big long blog post about how that works now. We've been doing it for about 9 months, but we're at the point where we can do it automatically and at scale. Really hard to express how much not having this capability increased our degree of difficulty.
* We're taming `flyctl`; in the immediacy, just doing a better job of reporting errors (it's a Go program, we love Go, but showing "context deadline exceeded" to a user is, obviously, a bug, albeit one we didn't recognize as such until recently).
I don't have a way of addressing these kinds of concerns, which are valid, without just being clear about what we're up to and how we're thinking. I wrote a sort of "once-and-for-all" thing about this mentality here:
I get the impression that the operations are a bit of a mess too, but that support is empowered to actually make people whole, which is a nice change of pace.
Compliance with what exactly? When you bring servers down to Brazil it gets weird, outside just not supporting that region, not much you can't do about it. We never did anything in Brazil because it was such a huge pain in the ass, it's hard to get stuff in and expensive to rack once you're there. imo Good for fly for doing the work, even if they did have to slip someone a 5 on the way in.
Compliance with ISO 37001 for example. As a big company, you can’t just go around and bribe random border guards to get your stuff through.
I’m also not sure if it is illegal to pay bribes in other countries when you’re an American company. So compliance with laws is another problem when paying bribes.
Looks like it is explicitly forbidden to bribe foreign officials as a US entity.
Not very smart to suggest you’re bribing people in other countries in official communications.
It's not overt, as they mentioned in their post, mystical fees just show up on your invoice from the government. I do agree tho, it's a really silly detail to include. I'm a bit passionate about this because I ran a long disability study at DigitalOcean to see if we could do Brazil.
Grease payments are allowed. You can’t bribe border guards to sneak things through, but if they demand a bribe to do their job or your servers get mysteriously blocked from import, paying them off is allowed.
> Despite efforts to provide advice, the distinction between grease payments and bribes remains a source of contention in FCPA enforcement, with legal experts and practitioners continuing to argue and interpret it.
Talk to corporate council before "grease payment"-ing anyone.
To be fair, Brazil does have weird and expensive customs/import duties on technology products, which makes a lot of hardware expensive. Maybe fly.io just didn't know about it and that's why they say "suspicious"; But yeah, it definetely sounds like admitting to bribery, which as an American-registered company is illegal even in other countries.
So, hosting in Japan is more expensive than regions in America? JPY is in a horrible place compared to USD. That seems like a good way to push away Japanese users away -- make it more expensive in countries where the local currency is weak.
This is painful, I was building for fly.io because of their dynamic request routing but I'm thinking I'm just going to go with hetzner. The cost difference is so substantial. If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
We're sorry. I hope we're clearly expressing what's happening here: either we do flat pricing globally, so that people deploying in cheaper regions subsidize those deploying in more expensive ones, or we reflect our cost basis in our prices and let our customers (and prospective customers, like you) make their own choices.
In your specific situation: it really just is the case that Brazil soaks us on import fees. Getting machines in racks in Brazil is just crazy expensive. We're going to keep doing it! Brazil has too many good sandwiches. But the idea behind building a new public cloud on our own hardware is for us to be around for the long haul, and flat low global pricing is not a "long haul" decision.
(It wasn't a long haul decision last year, either, but building a billing system capable of expressing this stuff is a nightmare from which we have not yet fully awoken. It turned out to be a shockingly hard problem.)
There are going to be cases where places like Hetzner make a lot sense compared to us. We hope you get your thing launched and find success wherever you end up. Some of our stuff, like LiteFS, you can take with you! :)
I think the change makes sense, even though I have workloads running on the regions that got the steepest price changes. I'd rather have more granular pricing than a blanket increase. The why now also makes sense - there was a technical limitation that doesn't exist anymore.
I was expecting a larger grace period from announcement to price change though. You got it right when you started charging for stopped volumes - you overcommunicated and gave users estimates for a few months before you started charging. Smoothing the increase over the next 4 months was nice.
A couple of nits as well: - AFAIK prices increased in all but two regions. The announcement could be clearer or more direct regarding this. - The pricing page could have a clearer from/to. I have to click on the example toggle to know how much machines cost before the change.
This same argument applies to most cloud hosting, non-cloud (or less cloudy, Hetzner is a little cloudy) will always be cheaper because the service is more basic in a number of ways.
Fly’s pitch is not just on demand scalable compute, but also that compute being close to the edge, and with “ops-less” deployments. All of those are factors you pay for.
You could pay a little less with a big cloud provider’s serverless platform, a bit less again with regular cloud VMs, a lot less with colo hardware, and a ton less running your own DC (at scale). Obviously not all of these work for all businesses, but saying Hetzner is cheaper than Fly misses the point of those two services on the spectrum of service levels.
> will always be cheaper
This has me so confused. Should the cloud abstractions not clearly make the service more optimizable at scale and thus cheaper to offer? Sandboxing things, limiting options, not having to deal with users wanting to provision VMs and do all kinds of unexpected stuff? The marginal cost of the beautiful software to make all of that happen smoothly, and all the dx it affords, that will just go towards 0, no?
I understand that people ask for what they can get away with. But does the market fail or am I missing a piece of the cost/price puzzle?
No it won't reach 0, for many reasons.
- Things change and break. You are paying for people to fix that for you at 2 AM, and to keep up with industry developments and best practices and make those decisions for you.
- Legal risk and compliance requirements change. You are paying for them to keep up with those for you.
- Overhead imposed by multi-tenancy. Security boundaries, control planes, etc. These are fixed operational costs caused by the abstraction.
You're not paying for just an abstraction, you're paying for the things the abstraction enables. It's also not always what you need. But it often makes good business sense.
Interesting take.
At one side, the big cloud providers have their very fair profit margins. But I've always assumed that comes mostly from that they can, and do, charge extra for the beautiful software (not seldomly through ridiculous egress bandwidth pricing).
I'm skeptical of how much flexibility and optimisation the big cloud providers really get compared to a very large VPS provider?
All the compute products (different form of container hosting, serverless) are essentially still priced at CPU+RAM + a premium. Some additional abstraction for nouveau databases ("capacity" or "billing" units). Many of the highly abstracted offerings also have the scale-to-zero concept, while the provider still have to have physical capacity ready (to some extent).
Sure, you can probably crank out some percentage more utilisation, but that doesn't really matter for the 100+% price-add on the market bears.
But I of course have no data to back up my claims, just speculation.
On a systems engineering level, keeping a system in a state of low entropy for a prolonged period of time will always be harder than a state of higher entropy. In practice, this comes down to more complexity in the software involved, and the people required to make the machine go boop will inevitably higher, since you need to maintain the servers and the complicated stack on them.
The tooling for managing the "lower" layers is progressively worse / slower / harder the closer to the metal you get, so you usually are paying a heavy premium for convenience / DX.
Only very specific serverless products where the provider can bin-pack you with other tenants allow them to possibly offer lower prices.
The sandboxing (by way of example) is a hardware affordance, not a software feature. In order to hardware virtualize every instance running on Fly.io, we have to provision hardware globally; we can't run customer jobs in namespaced userland containers under a shared kernel on cloud-provisioned hosts.
IMO it’s rather sad that virtualization with native-like performance isn’t available on ordinary cloud instances. Getting nested virt or some kind of hypervisor-assisted partitioning working is hard but far from insurmountable.
For a lot of people (most...?), Hetzner gives them exactly what they're looking for at an excellent price point. As an ecosystem, we never really moved past VMs, we just went up the stack more.
I'm not missing the point of the service, I'm against the idea of infrastructure being an 84% margin business. But I guess this is what the VCs demand and some are willing to pay it.
Don’t host anything important on Hetzner, as they’re cheap they get a lot of questionable users, and their systems are tuned to null route anything that looks remotely suspicious or high-volume to their hair-trigger network monitoring and DDoS protection.
And then you’re at the mercy of whatever support agent decides to look at your ticket and that can take days.
Use literally anyone but Hetzner is my advice.
I disagree, at my company (elest.io) we have several thousands VMs with Hetzner and reliability is very comparable to AWS from our internal stats. In case of DDOS attack we get notified and can discuss with their team.
Also support is competent and quick. We usually get an answer in few hours most of the time.
I’m glad you’ve had a positive experience, maybe it’s the usage level that grants you that treatment. As a one-time new client I had the worst possible experience with them: null routed my game server on launch day, very slow to respond support team, and effectively held my server hostage for 3 days. I guess I needed to inform them I was intending to use the server I paid for?
Took my data and never returned.
there is no way Hetzner is worse than fly for uptime. Fly has deleted my db, taken servers offline for hours, and expired tls certs for hours without telling me.
Absolutely not defending fly.io. OP is migrating off fly and in my personal experience, Hetzner is not a good option. Maybe OVH, AWS, GCP, Azure, Render, Digital Ocean, or linode.
I haven't heard bad things about DO in a long time, but a few years ago they deleted customer data in their cloud storage solution.
Years of excellent uptime with Hetzner here. Almost decades.
I disagree, the support is very responsive. Try to let them know in advance about your plans and you will be fine.
Adding another one to say that I've only have a positive experience with Hetzner so far. 6 months in, and 4 machines. All doing fine.
I've been using it for years with high bursty loads from users. Never had an issue with network stuff between my user and hetzner....
Hetzner has been fantastic for us for many reasons. I’m not convinced the problems you ran into are Hetzner specific.
How is it you can have a good experience, and yet my negative experience somehow doesn't qualify and you're insinuating the issue is with me, and not Hetzner?
https://www.trustpilot.com/review/hetzner.com <-- There are a large number of both 5-star and 1-star reviews with almost nothing in the middle.
This happens with every provider. One thing you're seeing is just that people with 5- and 1- star reviews have a lot of motivation to post reviews. People tend not to be motivated by "meh it's fine" feels.
> If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
Please give us an update here when you are done building your hetzner beowulf cluster and let us know how it fares. I think you are grossly underestimating the system you describe.
Whether this is completely out of the box or a multi-year engineering project to get equivalent levels of utility in the long term is completely up to how one specifically uses Fly.io.
E.g. a load balancer in front of some relatively stateless http microservices is really not going to tie you to Fly.io because of complexity. A globally distributed edge compute environment with database, RPC, varying scaling patterns, and environment certifications is probably not going to have the value prop impacted by this kind of minor pricing change though.
True, but managing 5 servers requires overhead. Where does the load balancer sit? What happens when it goes down? What if the entire datacenter loses comms? That is why cloud providers have AZ's that are physically separated from one another at different sites. When your boxes are spread across different physical networks, how do you connect them securely? For true HA you need a system that can self heal. Just throwing servers at the problem does not make things better. Reminds me of the expression 9 women can't make a baby in 1 month.
None of this is impossible, but when you are a small startup it makes sense to offload this to someone else and not waste time reinventing the wheel.
For the record, hetzner cloud also has multiple data centers (at least in Europe, haven't checked for US) with sharedand load balancers . What you don't get is managed services like k8s, databases, sqs, lambda/faas. These are generally easy to setup via preconfigured Terraform or ansible scripts, but if something goes wrong, you'll have to look at the logs yourself - that's true. (But that's ignoring that such a small team as you're proposing would likely be better served with a more mundane setup, avoiding all of these extra points of failure you'll have to manage... Making a simple bare metal setup once again the less involved and better performing alternative)
Hetzner load balancers don't have any auto scaling feature which is a pretty critical requirement for any load balancer product in 2024.
Im pretty sure you're vastly underestimating the amount of performance you get from today's hardware.
You won't be a small team by the time you're at a scale to need horizontal scaling. Only the biggest consumer facing/social media sites get value from that.
If you cannot tolerate downtime you need a redundant load balancer and you need nodes in different datacenters. Scale really doesn't matter in this equation. Now maybe you don't need five-nine's so then the risk is worth keeping it simple.
Both of which hetzner has? Did you skip the previous comment entirely?
I read it. Historically Hetzner has been popular in the dedicated server space. I cannot speak to their IAAS/cloud offerings, I have never used them. I will take them for a spin, though, seeing as it is really the only way to use their services in the US with good latency.
I don't want a cluster, I just want a load balancer that can connect users to the machine I want.
If the LB is for HTTP traffic, check out Cloudflare’s offering.
Hi, professional sysadmin here. I've run production deployments across hundreds of machines in Hetzner; it scales fine. The only real problem is that Hetzner is so much cheaper than anything else that it's easy to get a little bit stuck. (Honestly, that was a fun job in a lot of ways. Oh well.)
Never said it couldn't be done. I forgot that they have ventured into the iaas space and are doing more than just dedicated boxes now. For a small startup with a simple app, managing five dedicated nodes and building your own orchestration on top of deployments, load balancing, networking etc feels like the juice ain't worth the squeeze. For their cloud offering, it is more compelling.
I get that this stings, but honestly, outside of the brief outages when I'm trying to deploy for the millionth time, it's been a dream to run on Fly, with instances close to my users.
It's too bad the closest region to Dubai is Romania, and Africa/SE Asia are a bit uncovered, but that's ok for now. I'm really enjoying using LiteFS and globally distributed instances for each customer, making it really easy, fast, and scalable. As the sole dev on the team right now, this is fantastic.
I wish Upstash Kafka was a bit more baked, but I ended up just signing up with them directly. I had trouble using the built-in-to-Fly version, but that's fine.
Can't wait to give them $1000s a month.
> Did you know it costs us twice as much to rack a server in São Paulo?
There is an interesting dynamic of server prices and customer spend in different parts of the world.
Usually where the server prices are high, customers spend less.
There's also physical limits though, and structural economic lag in terms of investment. If you look at any CDN costs globally and compare it with the number of intercontinental fiber connections between those places you generally start having a good idea that the two are related. In some places there's also just less qualified talent (due to having less ability to train people in enough universities, or just not having them for as long) and datacenter space built and extra import costs that delay everything.
At some point everything will saturate and probably energy production costs will dominate (largely for cooling), but I think we're still a bit far from that.
In the case of Brazil it's mainly due to our absurd import taxes.
Do you have numbers to support it? If you mean import of hardware that do makes sense, but what about hardware that's already here? (I'm also Brazilian)
Looking at AWS site it explains that PIS/COFINS and ISS are charged, excerpt below:
"AWS SBL can issue an NFS-e to a Bra zilian customer only after receiving a valid Tax Registration Number ("TRN") (i.e., CPF/CNPJ). The relevant taxes applicable to AWS cloud services offered locally are the following: PIS and COFINS (9.25%, combined rate) and ISS (2.9%, charged by São Paulo municipality)."
https://aws.amazon.com/tax-help/Brazil/
This is below 15% which many countries do tax for cloud services.
I'm not saying that Brazilian taxes are really high, they are for hardware imports, but the taxes into the service itself does seem "fair".
> but what about hardware that's already here? (I'm also Brazilian)
It's ridiculously expensive.
Very little hardware is produced domestically, and the little that is, you don't want to be running your switches on what is in essence a no-name "intelbras" brand.
If you can help us drastically reduce our Brazilian hosting costs, we're all ears. :)
I won't deny that I'd love the challenge. However, there are some caveats:
1. AINAL, so I might say legal matters wrong for not understanding completely how the tax law works in Brazil, (Tho I do like to search about it), but I might find someone who could help you guys with it.
2. Currently, there is a law already approved (now being regulated with complementary acts) about how taxes will change in Brazil to the VAT model, this could increase/decrease taxes for a service like yours.
Besides these 2 points, we could talk via email (xunjin.coder@gmail.com) to might find a way like AWS did, creating a Brazilian subsidiary, to be able to generate NF-e in accordance for each valid Brazilian Tax Registration Number (CPF/CNPJ) and issue the invoices in BRL.
This page https://aws.amazon.com/legal/aws-sbl/ could have more info to elucidate how AWS does.
Seriously, thank you for offering. I'll let Scott and Dave know, and one of us will get back to you. (If anything comes of this, and you're OK with it, we'll write it up so people on HN can read about it; I always feel a little squicky "moving" discussions from HN to private channels).
Be my guest in writing about it, I hope I can help you with it.
PS: Life is sometimes a funny coincidence. My brother, who is a State attorney where I live, called me to share private matters 20 minutes ago, and already asked him for some help/directions about it. This weekend, I will speak to him in our spare time about it.
Great!, then I'd rather my app be a littler slower for those people if it saves me money. If the latency is not noticeable, why pay more to have that edge hosted.
Recently evaluated Fly.io for a company project but decided against it. I get the vibe that it fits the hobby/side project market more and that they're still in startup mode.
fwiw, we deploy on fly and are happy with it.
The reason to like fly is you get a 20-ish line yaml file that replaces (and I'm not exaggerating) thousands of lines of cloudformation. It automagically sets up the equivalent of a load balancer, autoscaling group, VPCs, db, vpn, etc. And it's trivial to replicate that environment per employee.
Plus the autoscaling using their firecracker stuff is fast.
I'd recommend most companies who aren't using tons of aws services take a hard look at fly.
They did remove the hobby plan too
It can still be used for free as from my experience and the docs, monthly bills under $5 are free.
Can you link to the docs you’re talking about? The docs I see say you need to have a legacy account.
I can’t find it in their docs but they stated it in the announcement email for pay-as-you-go. I also have received a email saying they covered my ~$0.18 bill for the month. You could probably contact support for more clarification.
Pretty funny that I built an app in Brazil and then this happens right before it goes live.
It would be nice to have stable pricing and not have a fear of different features costing more out of the blue.
Other than this, fly has been a great service, I love the system, the CLI and litefs.
It scares me that internal bandwidth charges will start being added, the application I’m using has a lot of internal transfer which is currently free.
Internal bandwidth price is free? I decided to avoid Fly.io because the pricing page suggests it is not:
"Any traffic exiting a Fly.io Machine is considered outbound data transfer, including:
- traffic destined for the internet - traffic destined for other Machines or apps in your network"
I have many edge nodes with data being synced to them internally with liteFS. This is free
Edit: maybe you are right actually. I have to check on this closer. App is just about to launch so I hadn’t paid so much attention
The nice thing about Fly is it’s very docker oriented, so depending on your app migrating it off may be fairly straightforward?
My apps are all on Fly but in the regions without big hikes thankfully, so I’m sticking around for now, but it sucks some people are being hit by this…
Technically you are right. However my app is using litefs. In theory you could set this up on your own servers. Which is part of the reason I was so interested in it.
However with fly it’s just one command to add a new region and my data is auto replicated there.
That’s super nice compared to manually managing this.
As long as bandwidth costs do not go up drastically I think it’ll be okay.
Fly is the only place I can get a lambda-like application with an attached SSD. Tigris is cool too (s3-alike but faster for our use case).
FWIW, I haven't had too many reliability concerns but I have had some issues with builder VMs (can always build locally so no big deal) or non-prod environments with just one instance.
Looks like they've quietly (?) deprecated the hobby plan too. Getting 3 instances a month for free was probably too good to be true forever..
Why do you think that? Still visible here: https://fly.io/plans/personal
I hope they don't cancel this plan, currently on it :D
EDIT: Ohh, it's only visible when I'm logged in, weird...
EDIT2: ...indeed
> The paid Hobby plan and the Legacy Hobby plan are not available for new sign-ups.
https://fly.io/docs/about/pricing/
damn :(
> If you were on the free Hobby plan at the time that the paid Hobby plan became the default for new organizations, your plan is now called the Legacy Hobby plan. Your costs stay the same as they were, with no monthly subscription fee, and no included usage beyond the free resource allowances.
so i get to use the legacy hobby plan because i'm grandfathered in, nice. only use it for a hobby project with one instance anyways(see bio), if they really decide to start charging me i'll gladly leave.
Heya, we deprecated pretty loudly! We made a post on the community forum [0] and we sent an email to all active accounts saying the following:
> The first improvement we're excited to announce is that the $5 Hobby plan is no more. We're replacing it with a very simple Pay As You Go plan. On this plan there's no more upfront $5 charge and no minimum monthly commitment. You only pay for what you use. If you don't use anything for a given month you pay $0. You still need a credit card on file to prevent abuse. But your card is only charged if you use the service.
[0]: https://community.fly.io/t/fresh-produce-pay-as-you-go-plan/...
If you're going to stand behind your product, I'm not sure posting with a throwaway sends the strongest message.
If anyone's ever wondering how carefully choreographed our HN presence is, please refer them directly to this subthread, which, for posterity's sake, I will note is occurring on the morning of July 4.
Sorry, didn’t intend for throwaway to come across as not standing behind the product.
I wrote the community forum announcement and was one of the people that worked on implementing our PAYG plan. I just wanted to clarify that we didn’t deprecate silently, so I quickly created an account to do so.
Apologies for any confusion this caused!
random side note, it would be nice if the region selector for regional pricing would display human readable region labels in addition to 3 letter acronyms.
yeah, it's basically impossible to compare regions right now
they need a table that's a human readable name and an average/range for the cost modifier as compared to the baseline
Why a throwaway?
I didn’t get this email. I don’t check the forums for announcements
I got three correspondence from fly on this matter when it happened. They absolutely were transparent and clear about it.
Same
Whoa, they did get rid of it. Not surprising in the least, but they definitely did it with a minimum of fanfare. When Heroku got rid of their free plans at least they were upfront and released a blog post about it. Maybe they did and I just didn't notice.
Fly.io and Render.com are alternatives to Heroku. How do the two compare in terms of maturity? Can they both be trusted with production apps?
We looked at both and chose Render based on various posts here along with our real world testing and bake off parameters. We use the Pro Ultra config and handle ~20rps on a Node / Express stack w/ 36 workers using clustering. Our bandwidth is around 50Gb / month.
Render has been quite solid and the support has been on point when we have found issues or run into unexpected edge cases. I have been impressed that despite a big raise and subsequent scaling up, they continue to ship a solid product and the platform improvements have been useful as well.
Never used Fly.io but I have a website running on Render (2.4M unique users per month) and the service is getting quite nice. Occasional hiccups, but they are improving. Not as stable as Heroku used to be 7-8 years ago yet, but it's also waaaaay cheaper.
I really like their Docker support and infra as code, makes it very easy to spin up a whole thing while not being too far conceptually from Kubernetes for example.
Running production app on fly. There have been some hiccups but generally we're quite happy with it.
I have services running on each. Fly's more flexible. Render's further from the metal, and thus somewhat more limited.
Both have some reliability issues and rough edges. My new stuff is mostly on Fly, but I'd use Render where I had a junior team who had zero infra chops.
Very happy with Fly.io. My website Pillser runs on it and I never had issues with them. The cost per month us just under USD 100 for hosting 3 services, including the database.
They currently cost 4 times more than Hetzner.
Hetzner gives you bare (or virtual) metal that you have to install stuff on, and maintain it.
Fly.io and similar give you a higher abstraction layer where you throw a Docker container and the stuff below and around it is managed for you. Basic suff like OS patches, but also autoscaling, load balancing, request routing, etc.
Does the 4x difference make it up? It will depend entirely on you and how good you are with the things fly.io abstract for you, and how much you value your time spent on it.
If you are literally just looking for a place to cost-effectively park a VM, you should very seriously consider Hetzner! It would be weird if there weren't use cases where Hetzner comes out ahead.
HN always mentions Hetzner, but they're no longer an exceptionally cheap VPS provider. They're basically the same price as OVH.
I forget when exactly they raised their prices, but it was several years ago.
The big benefit of fly is that your server only costs money when it’s doing something (with autoscaling enabled).
> our billing system has been hot garbage for most of the life of the company
I’ve heard the same thing about their core product.
Oh wait, they even acknowledged it: https://community.fly.io/t/reliability-its-not-great/11253
Don't know if anyone noticed, but Ben Johnson recently became head of product over there. I'm fortunate to have called Ben a friend for a long ass time now and I gotta say, I'm extremely bullish on fly as a result of this. Ben is not just a developers developer, he's genius level intelligence. https://github.com/benbjohnson
(We fumbled the ball hard at DigitalOcean ngl, fingers crossed Ben can build what I dreamed of building one day!)
What/why do you think DO fumbled? Fly looks like what DO could have been, and they had a 10 year head start.
I'm curious as well. Pretty happy with DO overall; but it is relatively vanilla as far as a hosting product goes.
I’ll be bullish on fly when the awful outages stop. They had a hardware outage in the LHR region for a month, it was probably longer but I moved to a different provider so I haven’t bothered to check.
https://fly.io/infra-log/
How has Flys reliability been as of late? From past discussions I got the impression that despite it being quite polished on the surface, behind the scenes their operations are a bit of a mess.
e.g. https://news.ycombinator.com/item?id=36808296
https://news.ycombinator.com/item?id=39365735
From personal experience, still not great. Sometimes things just don't deploy.
I host my little mobile braai simulator on it (braaisim.com) because it's an art project and easy to add new domains / certificates. Convenient to be able to scale up easily if it went viral in ZA. Nice that they support Johannesburg region (jnb), though, which is close to me.
You were downvoted, but as a user, I’ve experienced the same inconsistency with deploys.
There’s also a hard container size limit I’ve run into multiple times. If you add a dependency and go over the size limit, your app won’t deploy unless you switch to a GPU instance, which is substantially more expensive.
Don't switch to GPU instances just to get around the container size limit!
What are you trying to do and what's the size of your container? My instinctive response here is that you're not holding it the way we expect you to: that some of what you're sticking in your container, we'd expect you to keep in a volume.
One thing you can do to minimize container size is have multi-stage builds in your Dockerfile so that you are not affected by cache bloat.
I see two sibling comments with reports of patchy reliability, so I'll note that it's been rock-solid for me (multiple daily deploys every weekday) since I last commented on it (about a year ago?)
A bunch of mini outages and the main site + dashboard was out for maybe an hour last week on Thursday.
my production app has had 1 outage where stripe webhooks weren’t delivering for 45 minutes which is a huge issue for sure.
I mitigated by checking in addition to the webhook event and also it hasn’t had a problem since. Their explanation was a 3rd party networking issue at or near one of their data centers
Bad enough that we had to migrate our key customer-facing platform off of it because of the impact.
I love the idea and will still use it for many less-business-critical apps but after multiple late-night weird un-debuggable database outages we had to switch.
To provide at least one positive example, I'm using fly.io to host a website I made for a friend (for her college capstone project), so I'm on the now-legacy hobby plan paying nothing, and I have had zero problems.
They upgraded my server and it was down for about 8 hours. They warned me about the upgrade, but didn’t tell me they failed to restart my server.
The very first time I tried to spin up a test deploy the instance never came online, which I thought was pretty funny. This was like 6 months ago.
I use it to power CI builds (a lot of them!) and have extremely little issue with it.
Basically I’m just using the API to spin up machines, which do some work and shut down. There’s some extra machines per build job, like database containers or a headless browser for testing. Pretty smooth in my experience.
I think the only occasional issue I hit is the internal DNS being a few seconds behind reality.
You're not entirely unbiased about this. :)
https://fly.io/infra-log/
Detailed description of every incident (a superset of "outages"), including stuff not visible enough to make the status page, going back to April.
Semantics aside, if an incident/outage/<other term> affects devs/users in any way, it really should be on the status page.
I found it impossible to distinguish between user error and platform outage. Too often it was a problem on fly's end yet the status page gave nothing (perplexed, I'd rerun the deploy a few hours later and it would work).
Can't stress it enough: if fly's services aren't working for any reason, big or small, put it on the status page. Devs need to know when something's not in their control so they can inform their team and customers and go to bed or take a coffee break. They shouldn't be up till 5am thinking they borked something when the problem is 100% on fly's end.
We have the same semantics. The incidents that are on the infra-log and not on the status page are things that didn't affect devs/users in any way. A good example: we have clusters of edge servers in every region we serve. They're servers, part of their job is to occasionally throw a rod. When that happens, we pull them out of service and stop advertising them. That's an internal incident, but it's not a status page incident. Preempting a response here: no, I don't think users would be better off if we status paged stuff like this.
Every week, we go through the incident management system, and all our incidents, whether or not they had user impact, get written up here.
One thing you're probably running in to, which is not on you but rather on us, that we are aware of, and that we are grinding steadily away at: as the platform has stabilized over the last year, an increasing share of problems users run into aren't "platform" issues (in the sense of: things going wrong in our API or on our cloud servers) but rather bugs in `flyctl`, our CLI.
We have two big reliability projects happening:
* We've finally landed our "white whale" of being able to move workloads that have attached volumes between different physicals. I'm writing a big long blog post about how that works now. We've been doing it for about 9 months, but we're at the point where we can do it automatically and at scale. Really hard to express how much not having this capability increased our degree of difficulty.
* We're taming `flyctl`; in the immediacy, just doing a better job of reporting errors (it's a Go program, we love Go, but showing "context deadline exceeded" to a user is, obviously, a bug, albeit one we didn't recognize as such until recently).
I don't have a way of addressing these kinds of concerns, which are valid, without just being clear about what we're up to and how we're thinking. I wrote a sort of "once-and-for-all" thing about this mentality here:
https://news.ycombinator.com/item?id=39373476
I get the impression that the operations are a bit of a mess too, but that support is empowered to actually make people whole, which is a nice change of pace.
> which is a nice change of pace
That’s about every early stage company, though.
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
Compliance with what exactly? When you bring servers down to Brazil it gets weird, outside just not supporting that region, not much you can't do about it. We never did anything in Brazil because it was such a huge pain in the ass, it's hard to get stuff in and expensive to rack once you're there. imo Good for fly for doing the work, even if they did have to slip someone a 5 on the way in.
Compliance with ISO 37001 for example. As a big company, you can’t just go around and bribe random border guards to get your stuff through.
I’m also not sure if it is illegal to pay bribes in other countries when you’re an American company. So compliance with laws is another problem when paying bribes.
Edit: https://en.wikipedia.org/wiki/Foreign_Corrupt_Practices_Act
Looks like it is explicitly forbidden to bribe foreign officials as a US entity. Not very smart to suggest you’re bribing people in other countries in official communications.
It's not overt, as they mentioned in their post, mystical fees just show up on your invoice from the government. I do agree tho, it's a really silly detail to include. I'm a bit passionate about this because I ran a long disability study at DigitalOcean to see if we could do Brazil.
I suppose you meant "feasibility study" or is "disability" the actual word?
Oh, that makes a lot more sense lol
What does a "disability study" mean in the context of international server installs?
Look at the Exception section.
Grease payments are allowed. You can’t bribe border guards to sneak things through, but if they demand a bribe to do their job or your servers get mysteriously blocked from import, paying them off is allowed.
> Despite efforts to provide advice, the distinction between grease payments and bribes remains a source of contention in FCPA enforcement, with legal experts and practitioners continuing to argue and interpret it.
Talk to corporate council before "grease payment"-ing anyone.
To be fair, Brazil does have weird and expensive customs/import duties on technology products, which makes a lot of hardware expensive. Maybe fly.io just didn't know about it and that's why they say "suspicious"; But yeah, it definetely sounds like admitting to bribery, which as an American-registered company is illegal even in other countries.
So, hosting in Japan is more expensive than regions in America? JPY is in a horrible place compared to USD. That seems like a good way to push away Japanese users away -- make it more expensive in countries where the local currency is weak.