Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure runs your company. Trying to outsource it is like trying to outsource upper management.
To be clear, I'm not saying that the optimal amount of cloud services for an established company like FedEx to buy is 0. They bring in management consultants, too. But it sure isn't 100%.
That sort of guy. Well he once told me something about executives and upper managers working for corporations that I have never forgotten. He said to me, and of course I am paraphrasing:
"Change gives the illusion of progress". I asked him what he meant and he responded with something to the effect of "They have the habit of changing big things every 5-10 years on purpose to make it look like they are productive, and to justify their own roles, one guy will come in and 'cut costs', the new guy after him will 'invest'".
Well, six months later sales and profits were still way down and the new CEO was catching a lot of heat. He began to panic but then he remembered the envelopes. He went to his drawer and took out the first envelope. The message read, “Blame your predecessor.” The new CEO called a press conference and explained that the previous CEO had left him with a real mess and it was taking a bit longer to clean it up than expected, but everything was on the right track. Satisfied with his comments, the press – and Wall Street – responded positively.
Another year went by and the company continued to struggle. Having learned from his previous experience, the CEO quickly opened the second envelope. The message read, “Reorganize.” So he fired key people, consolidated divisions and cut costs everywhere he could. This he did and Wall Street, and the press, applauded his efforts.
Another year passed and the company was still short on sales and profits. The CEO would have to figure out how to get through another tough earnings call. The CEO went to his office, closed the door and opened the third envelope. The message began, “Prepare three envelopes...” "
IN PARTICULAR, at one Fortune 250 (which no longer exists), we implemented OneWorld to replace the mainframe. After 7 years, we still had the mainframe, AND a badly-implemented version of OneWorld. Then the company got bought by another Fortune 250, moves were made to "commonize" the IT systems, and then the parent company sold everything. But I'm positive that everyone involved in the project to retire the mainframe made the project look very successful on their resumes.
I don't know if the above fear would actually play out, nobody is willing to not make changes to find out.
High level management is fundamentally hard to stay on top of over time. It's about as easy as thinking chess is easy because you can theoretically know every move your opponent might make. There are so many moving parts to an organization that having visibility to them isnt enough to perform well. They have to influence most of the business pretty indirectly. If changing things gives you the confidence necessary to keep things running.. that's what you're going to do. Everything is a gamble, so doing nothing is kind of unacceptable leadership behavior unless they are actively taking up the mantle left by previous management and understand it very well already.
It's hard to keep a good (confident, ambitious) team from re-engineering. All the same dynamics apply: In your mind the disadvantages of change are small because you don't know them, but the advantages are large because you planned them. Making change gives you more control over your fate because you are executing your own plan as opposed to staying the course. Finally, how do you keep people motivated to show up every morning if you don't have a vision for change in the future?
I don't think its that different for managers and engineers. There's a lot pushing people to try something, even if the objective odds of success aren't great.
Management are contemporary clergy, spewing high minded ephemera, only to go home unable to point at anything net new left behind by their effort.
I grew up in farm land; we had no middle managers. Somehow food still got grown, harvested, and sold; somehow a Linux kernel and other wildly popular open source exists without them.
Post-WW industrialism needs to wind down. Militant minded people came home and forced their PTSD on workers. We spend a lot of resources equipping people to output nothing in deference to traditional economic memes. America of recent decades necessarily built itself into a production powerhouse to resupply a destroyed world. Such memes are outdated given automation and unsustainable given real material costs.
It's an area that is pretty tough to make meaningful criticism. its kind of a show dont tell type situation imo. If upper management seems under-qualified and overpaid to you, then maybe you have a calling to go perform better and get paid more.
Company management is in underdeveloped game-theory territory. Sure we can isolate one part of their job and describe how they are failing on it, but we dont know the trade-offs being made on a daily basis with their time and focus. a lot of which is going to be company secrets, if it even leaves their personal thoughts. Any criticism that comes down to saying "they should have done more" is likely out-of-touch, for example. Unless you can prove they were actually being lazy.. which is usually not the case, since they are often workaholics (ime). But we usually cant tell if something is a good move or not until it plays out on the market. So making criticisms based on hindsight is weak, as is making criticism that lack the full picture of the organizations goals and the time / energy they actually have on hand to accomplish them
CEO is a very general job role. I dont understand your point with the painters. That would be an operations issue, so I actually wouldnt criticize the CEO of the painting company at all for it. thanks to him/her, I was able to contact a painting company, they showed up, they painted the apartment, and left. I would think the operations team (the painters) deserve to be fired and held accountable for claiming they know how to paint a room when they clearly didn't. Nothing about their job is general or consists of trade-offs. If I said "you only have 10 minutes to paint the room and then leave" then yeah, it might come out like shit and the "excuses" would be valid. which is the kind of time pressure CEO's are often under with respect to things they are actually doing on a day-to-day basis.
i would hold the CEO responsible with respect to resolving the issue and refunding me, etc, since the CEO role is to be responsible for the outcomes of the company.. but he's not the one painting the room. Just like with any company, the CEO does not literally run the company. If service is poor, it is usually because people are finding their way past hiring filters to get jobs they aren't qualified for. Let's not forget that people everywhere are often advised to lie their way into employment, fake it till you make it, baffle them with bullshit, reword your resume to sound more impressive, etc. These people line the mechanisms that CEO's use to accomplish anything at any company.
Of course, the CEO position is no exception to this and I am not saying it is literally impossible to build a case against a CEO. I am saying it needs to fully encompass the position or else you're likely assigning criticism to the CEO for some culmination of lower level operational incompetence that they simply failed to overcome. If a director over-promises to the CEO and the CEO signs off on the basis of trust with the director, then when the bar is not met later of course the CEO will be held responsible but the reality of fault sits with the director, or maybe a subordinate to the director who convinced the director that the over-promise was doable. You then have to get into the weeds of whether or not there were signs the CEO should have seen as to not trust the director, or if they had reason to overrule the directors approach, etc. you then have to do similar things across all areas of the company to derive a valid criticism that the CEO is the common denominator in it all.
Leadership is significantly harder to criticize appropriately than operations. Personally, I would like to stop reading meaningless criticisms from people who want to complain and be heard but dont want to do the work necessary to make a valid complaint.
So if had a horribly painted room that you just paid extremely well to have completed, and the painters came up to you and gave you a laundry list of reasons that they failed... Would you hold off on criticizing them until you had a complete understanding of what it takes to be a painter?
trying to use painting as an analogous situation like that isnt transferable to the point i am making though. Putting the excuses in their mouth doesnt even make sense. We are presupposing that the painters did a horrible job.. while discussing how to decide whether or not a CEO did a horrible job. The only reason you know the painters did a bad job is because we are saying they did. the only reason we can use painting as an example is because most people can imagine a terrible paint job. i.e. we do have a full scope understanding of what it takes to be a painter. I am saying it is much harder to imagine the role of a CEO and what good results would look like than it is a painter.
Maybe my wording was fuzzy, but I am not saying you need a complete understanding of the CEO's role, but it does need to be of full scope. I see that reads near synonymous, so in other words it may be infeasible to account for the total depth of their role, but at the very least the entire breadth of the role should be looked at. If you default to "i gave you a lot of money to make it happen so it should be perfect" type logic; you're just being a "karen". the cost of something has nothing to do with the results, directly. Money needs to be converted into something that helps the work, and in that process we are all still limited by reality; diminishing returns, supply chains, quality of communication, availability of resources, etc. A CEO is at the focal point of all of this, and is human. Whether they get paid nothing or everything doesnt change how effective they can reasonably be.
But they do deserve criticism. it just needs a lot of work to do it right. you have to provide some sort of evidence that across all scopes of work the trade-offs do not make sense. maybe the CEO sacrifices on every front in order to provide the fastest service in the business and is successful in that. If you leave speed of delivery out of your criticism it becomes a meaningless criticism. "They charge a lot for poor quality". "These painters did a terrible job.. (even though I called them this morning, and they were done by lunch which allowed me to do a walk through with a potential tenant)".
All i'm really trying to emphasize is that we absolutely can criticize a CEO, but if you dont do it properly it is very easily washed away by the many unknowns of the position. however, if it is done right - it would be very damning as they cant default to company policy or directives from above as a scapegoat since they are the ones creating such things.
This is a bullshit reason based on jealousy, not reason.
> when those actions can have a negative effect on those below them and even to the external environment
This is the real reason it’s fair to be very critical of their maneuvers.
Reality is not zero sum, but neither are resources infinite. Is it really bullshit to critique more thoroughly that to which more of the finite resources are dedicated?
this kind of aligns with my point tbh. We give $10 critiques to million dollar positions as if they hold weight.
If this is trully BS, then allow lead developers to write checks for their whole team using the company's bank account.
they hold power in organisation, they can increase their own renumeration in a way that's rank and file staff cannot. Executive compensation has skyrocketed in the past 20 years.
If their management is ineffective, then they don't deserve top comopensation.
Satya: 1992, CEO in in 2014
Tim: 1998, CEO in ~2009
Sundar: 2004, CEO in 2015
Andy: 1997, CEO in 2021
Ted: 2000, CEO in 2020
They were all senior executives well before assuming their CEO roles.
That's a very succinct way to put it. I think that observation also applies to consumer technology (e.g. regularly re-doing UIs to "improve" them when the changes are either in fact regressions or just different but not better). We've had it drilled in our heads throughout the modern era that new == better (e.g. the ubiquitous "new and improved!" marketing language), but that's not actually always true. Change for change's sake justifies itself through that misunderstanding.
In the recent past, we had a really high rate of genuine technological progress. But at some point we'll have picked most of low hanging fruit and will enter a period of slower progress, where faking progress will become more an more tempting for producers than the real thing.
Depending where, the real-estate and energy prices are nuts in most places. And, the engineers are expensive right now, and the services are cheaper.
It's not about giving it up, or change for the sake of change, it's about seeing the writing on the wall. These managers see the larger trends, rising energy costs, maintenance costs rising, hiring difficult, and retention of existing engineers impossible. Once you see those trends and a department is underwater and it's getting worse, you have to move. At another point in the market, when you may find engineers are less expensive, easier to hiring, technologies and space are less costly and easy to deploy. You move back.
But how will the cloud providers avoid these trends? They won't. They will have to do the same thing as anyone else: pay more. And therefore charge more. There are economies of scale, but those savings are logarithmic and a company like FedEx is already pretty far out on the X axis.
AWS and Azure probably run hundreds of Fedex-sized clouds and it's their core business. I don't think Fedex is very far compared to them.
Innovations are more likely to happen if it is someones priority to fix a certain thing. I think they hope that savings from innovation and better methods at what ever company they hire out to are passed onto them. It is naive if they are unable to change clouds though that they will see these savings and as long as one relies on vendor specific features on is in that position.
Things that aren't Fedex's core competency.
Or, Microsoft's roofless datacenters, or locating datacenters in remote and colder climates, things the big players can do with economies of scale beyond buying things cheaply, they can customise the entire datacenter. Microsoft has experimented with underwater datacenters and modular containerised datacenter extensions which could be datacenters no human needs to be near to work on, or which could be dropped off somewhere with cheap land and power and internet, and picked up three years later and retired from use, or etc. Ideas which are not FedEx's core competency and need large scale and software clustering on top.
While FedEx would be hiring ordinary IT employees to work in a standard datacenter in cheap business park - not very enticing - Amazon could be hiring datacenter workers to work with Amazon's undersea cabling connecting their worldwide datacenters; more enticing work for skilled employees.
Google has been known/rumoured to migrate heavy batch processing workloads around the planet, following the day/night cycles to take advantage of regional cheaper night-rate electricity all the time. Something which reduces their energy costs but which FedEx may not be big enough to do.
But The Facebook decided to make informatics their core competency, to the point of building their own servers with 12 volts running to the rack, same as Google before them.
There were surely industrial companies in 01922 who decided that management wasn't their core competency (though they used different words), and if they needed help with management they'd contract out to management specialists like Taylor or Gilbreth. They met the same fate that will meet companies today that decide that informatics isn't their core competency.
So where are all these mainframe engineers going to go work? Our company has 3 admins for our two mainframes, and these guys while expert level admins for a mainframe, have trouble with Linux and Windows. Same with the developers writing code for the systems. When all you've worked on is a mainframe, then everything looks like a batch cycle...
That's deep and 100% makes sense, I can see how the cloud is both of these "things."
In exchange I offer two relevant quotes from my quote file:
> The empire long united must divide, long divided must unite; this is how it has always been. (Luo Guanzhong)
> When cuffs disappeared from men’s trousers, fashion designers gave interviews explaining that the cuff was archaic and ill-suited to contemporary living. It collected dust, contributed nothing. When the trouser cuff returned, did it collect less dust and begin at last to make a contribution? Probably no fashion designer would argue the point; but the question never came up. Designers got rid of the cuff because there aren’t many options for making trousers different. They restored it for the same reason. (Ralph Caplan)
That sounds ... dystopian.
Do you think it's "don't change"? Cause that gives right up on progress anyway. So good luck on getting large shareholders to give you the reins with that message. They're looking for ROI, not the status quo - if you don't evolve, your competitor will, barring monopolies and such. And even there... Microsoft of today is much less dominant than the MS of 25 years ago, and arguably could be worth a lot more had they made better moves. If that's not a compelling example, maybe check out Sears...
Maybe most people in these positions know that change doesn't guarantee improvement, but they know that sitting still is the same as just waiting to be defeated. So maybe there's something less-than-stupid about these "short-sighted" "illusory" changes.
But maybe if you want to be a wise executive, the key is to recognize that change might not just fail to improve your position - it might actually actively harm it. So the good executive is the one who chooses to try to change things according to reasonable calculations about both potential upside benefit and downside risk...
This is one of the biggest reasons many jobs are so miserable. You want to be in the job at the beginning of the"invest" decision. Unfortunately most people get little say in when they join because the information about this is kept internally to the company.
That itself is based on the belief that (historical) progress is an improvement.
Mostly true over the last few centuries - aspirin is nice - except for those occasions where it was mass murder.
I'm curious whether the folks claiming that have any data center ops experience.
Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no amount of money would make me work in a DC again.
I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.
Also, that's just compute. What about data? Sure cloud accepts your data cheaply, but they also charge you for egress of that data. Yes you should have your data in more than one location, but if you depend on just cloud then you need it in different AZ which costs even more money to keep in sync and available for training runs.
I think for simple workloads and renting compute for a startup, cloud definitely makes sense. But the moment you try to do some serious compute for ML workloads, good luck and hope you have deep pockets.
The moment the model gets to be bigger than the size of any one GPU's VRAM, the higher by orders of magnitude of difficulty in the process of training that model.
Here's the list of ingredients for Excedrin Migraine:
Active Ingredients: Acetaminophen - 250 mg (Pain reliever), Aspirin (NSAID) - 250 mg (Pain reliever), Caffeine - 65 mg (Pain reliever aid)
Inactive Ingredients: benzoic acid, carnauba wax, FD&C blue #1, hydroxypropylcellulose, hypromellose, light mineral oil, microcrystalline cellulose, polysorbate 20, povidone, propylene glycol, simethicone emulsion, sorbitan monolaurate, stearic acid, titanium dioxide
This is the list of symptoms that Excedrin Migraine claims to treat:
And now here's the ingredients for Excedrin Extra Strength:
Active Ingredients: Acetaminophen - 250 mg, Aspirin (NSAID) - 250 mg, Caffeine - 65 mg
Inactive Ingredients: benzoic acid, carnauba wax, FD&C blue #1, hydroxypropylcellulose, hypromellose, light mineral oil, microcrystalline cellulose, polysorbate 20, povidone, propylene glycol, simethicone emulsion, sorbitan monolaurate, stearic acid, titanium dioxide
This is the list of symptoms that Excedrin Extra Strength claims to treat:
- a cold
- premenstrual & menstraul cramps
- muscular aches
And while SOME places have normalized the prices between the two, they can be often found on shelves at two different price points.
The part that’s always missing with these rent vs buy analyses on HN for some reason is that it’s totally ignoring the opex cost of operating your own hardware which is going to be non 0. Sure, it won’t be quite as expensive (no profit margin) but it’s not an order of magnitude. Additionally, most companies don’t run the HW 24/7 and, if they do, it’s not a level of people they want to hire to support said operations. Not just running it, but you have to invest and grow something that’s not a core competency to get economies of multiple teams loading up the HW.
If the next revolution in cloud comes in to cause companies to onsite the HW again, it’ll look like making it super easy to take spare compute and spare storage from existing companies and resell it on an open market in an easy way. Even still, I think the operational challenges of keeping all that up and running and being utilized at as close to 100% as possible and not focusing on your core business problem will be difficult because you won’t be able to compete with engineering companies that have a core competency in that space.
Effectively hiring, retaining, evaluating and rewarding competent staff is hard. Even at a big company the datacenter can be a really small world, which makes it hard for your best employees to grow. Things are especially hard when you don't have a tech brand to rely on for your recruiting, and the staff's expertise is far outside the company's core business, making it harder to evaluate who's good at anything.
I'm not sure why you think that. AWS hasn't budged on their egress pricing for a decade (except the recent free tier expansion), despite the underlying costs dropping dramatically. GCP and Azure have similar prices.
Fact is, egress pricing is a moat. Cloud providers want to incentivize bringing data in (ingress is always free) and incentivize using it (intra-DC networking is free), but disincentivize bringing it out. If your data is stuck in AWS, that means your computation is stuck in AWS too.
I think we’re going to put real pressure for traditional object storage rates to come down. Since Cloudflare‘a entire MO is running the network with zero egress. As we expand our cloud platform it seems inevitable that you will at least have a strong zero egress choice and if we do a good job Amazon et all will inevitably be forced to get rid of egress. Matthew Prince laid out a strong case for why either scenario is good for us in a recent investor day presentation (either we cannibalize S3’s business and R2 becomes a massive profit machine for us because they refuse to budge on egress or Amazon drops egress which is an even larger opportunity for us).
Products like Cache Reserve help you migrate your data out of AWS transparently from any service (not just S3) - you just pay the egress penalty once per file.
Anyway. I’m not saying it’s going to disappear tomorrow but I find it hard to believe it’ll last another ten years.
Early in my career I worked at a company and we had a DC onsite. I remember the months long project to spec, purchase and migrate to a new, more powerful DB server. How much that costed in people-hours, I have no idea. I upgraded to a better DB a couple months ago by clicking a button...
Don't even get me started with ordering more SANs when we ran out of storage or the time a hurricane was coming and we had to prepare to fail over to another DC.
Compute costs generally drop over time. Do you have any data points to confirm egress will soon go to zero?
Data center ops is ultimately a supply chain management problem, but most people don't treat it as such. That was my primary learning from doing data center ops at a few different companies. If you get the supply chain management right, and are technically competent, there can be a lot to recommend running your own data centers.
If the AC breaks at 3am it neds to be fixed. It doesn't matter if you have your own HVAC people on sight 24x7, your own people on call to service it, a local HVAC service to come in, or you outsource the entire operations and so you have no idea how that is handled. In the end the important part of this story is that whatever you are doing with the AC continues to work. Different operations demand different levels of service (I doubt that anyone keeps HVAC techs on staff 24x7, but if the AC is that critical it is mandatory). The only case where the CEO is up at 3am is if the CEO is the owner of the local HVAC service company, not the CEO of the building with the problem.
Once you realize that to management the cost is outsourced no matter what the only question is do it with your own people and HR, or outsource it. There are pros and cons to both approaches, but for most companies it isn't their business and so the only reason to do it in house is they can't trust any company they hire.
The downside for saving the costs is that you're losing control with every step taken away: as soon as you go into a datacenter of any kind, you simply cannot call up a HVAC company and offer them 100k in hard cash if they're showing up in the next 60 minutes and fix the issue. With a colo DC you can usually go and show up there to see if the HVAC, UPS and other systems are appropriate to your needs, but with one of the big cloud providers you have to trust their word that they are doing stuff correctly.
Now I'm deeply confused. Any company hired either has a profit margin (plus enough to fund an "Oh shit" fund in case times turn bad) or will not stick around longer than a few years. At which case why not just hire people directly and cut out the other company's profit margin? Assuming you hire similar people at the same rate, using your own existing and already-paid-for HR, how is that not cheaper?
In some cases you can even get a discount. Utilities are. Big customer of tree trimming, the companies doing that work can give a great deal because the utility doesn't care that they take a week off after a storm for high profit margin consumer trimming.
Larger hotels often have dedicated staff for things like HVAC, etc, because the importance of getting things fixed quick if possible is worth the cost of having someone onsite/available.
And you see similar things with colleges, etc; they often have a maintenance deportment that can be pretty large (though no doubt they've spun it off and brought it back in-house for the same "change is progress" reasons).
There is a cost to taking on things that aren't part of your core business too.
Similarly, Google doesn't ship enough stuff worldwide to justify drivers, insurance, trucks, jets, etc. - Fedex has the size and scale to make every package a couple cents cheaper, so it's just not worth it for Google.
The only other argument I can think of is the challenge of keeping every plate spinning, in good times and in bad. This is where your point of having a cost to take on something outside your core business comes in, but we seem to be in an era of mega-corporations - I'd expect lots of companies to snake tendrils into whatever will save them a fraction of a cent every time they have to do something.
If your compute installation is big enough that payroll is a small fraction of the operating cost, then it's way cheaper than cloud. (that payroll has to include people who actually know how to build and run a huge compute installation)
The problem is that people come in integer units, you need a bunch of them to cover a bunch of different areas of expertise, and the particular ones you need are expensive. If you've got $1M worth of computers, you're almost certainly better off scrapping them and going to cloud, although the folks you're currently paying to run them might disagree. If you have $100M+ worth of machines it's a whole different ballgame; I'm not sure where the exact crossover is.
Note - that's assuming a single data center, and that you're big enough to build your own data center instead of renting colo space. If you need your machines to be geographically dispersed, you'll need to be even bigger before it's cheaper than cloud, and I'm not sure whether you'll ever hit crossover if you're renting colo space.
If you are sophisticated enough to engage an ODM, build your own facilities, and put hvac and electricians on 24-hour payroll, go on-prem. Otherwise, cloud all the way.
Like most startups, our management of the data center was pretty scrappy. Our CEO liked that kind of stuff, and we had a couple of network engineers that could be on call to fix overnight issues. It definitely wasn't a burden at the 50 employees size of company (and that includes field techs that actually installed fiber, dragged cable under the street, etc.)
We actually had some Linux servers in the datacenter. I don't know why, to be completely honest.
So overall my thought is that maybe use the cloud for your 1 person startup, but sometimes you need a datacenter only and it's not really rocket science. You're going to have downtime while someone drives to the datacenter. You're going to have downtime when us-east-1 explodes, too. To me, it's a wash.
AWS almost certainly gets batches of bad hardware too. And if your services are running on the bad hardware, you can't have a peek inside and find the iron filings. For servers, this is probably not too bad, there used to be articles about dealing with less enthusiastic ec2 vms since a long time, and if you experience that, you'd find a way. AWS has enough capacity that you can probably get vms running on a different batch of hardware somehow. With owned hardaware, if it was your first order of important database servers and they're all dodgy, that's a pickle; HPE probably has quick support? once you realize it's their hardware.
If your cloud provider's network is dodgy though, you get to diagnose that as a blackbox which is lots of fun. Would have loved to have access to router statistics.
There's a lot of stuff in betwren AWS and on-prem/owned datacenter, too.
I imagine the entire sentiment of the comments is because FedEx is one that really should be sophisticated enough.
There is a smooth curve between cloud and dedicated DCs, which has various levels of managed servers, co-location, and managed DCs. (A managed DC can be a secure room in a DC "complex" that shares all the heavy infrastructure of DCs.)
Primarily, the FedEx managers are committing the company long-term to Oracle/Microsoft platforms. Probably mostly to benefit their own careers.
Outsourcing hosting and management of DCs would have been something different, and probably healthier for FedEx and the industry.
You bet it does! But as the AWS customer you'd never notice because some poor ops dude in AWS-land gets to call up the vendor and bitch at them instead of you. It ain't your problem!
That doesn't sound like it will save much money, honestly.
And no, operators don’t disassemble to perform QC. And no, I could hire an entire division of people buying servers at Best Buy, and disassembling them, and stress testing them, and all of that overhead including the fuel to drive to the store would still clock in under cloud’s profit margin depending on what you’re doing.
You’re of course entitled to develop your cloud opinion from that experience. That’s like finding a stain in a new car and swearing off internal combustion as a useful technology, though, without any awareness of how often new cars are defective.
You can point at Google and speak in abstracts but it doesn’t address the point being made, nor that your rationale for your extreme position on cloud isn’t as firm as you thought it was. Is Dropbox the only time you’ve worked with hardware? I’m genuinely asking because manufacturing defects can top 5% of incoming kit depending on who you’re dealing with. Google knew that when they built Platform A. The lie of cloud is that dismissing those problems is worth the margin (it ain’t; you send it back, make them refire the omelette, and eat the toast you baked into your capacity plan while you wait).
You can test your stuff and be still profitable henzer aws etc would make no money otherwise....you know they test their server much more (sometimes weeks/month)
But i don't think they made even a initial load-/stresstest.
Unpack it, trow it into the rack, no checking of internal plug's just nothing...pretty sure about that.
I am beeing smug about not testing your hardware as you do it with software....shitty testing is shitty testing, counts for software hardware firmware and everything between. Even for your diesel generator ;-)
Load and simulated power failure tests all passed.
Then some time later there was a total power cut and that's when they realised the generator had an electric start wired to the mains supply.
> had an electric start wired to the mains supply.
But that's a good one, humans being humans...but it worked every time before today ;))
Do you see how these costs all start to add up?
Is that like too much these days for companies that owb fleets of cars? is opening a server harder than checking whars wrong with a car? like a cable comes loose and that's gane over?
So you don't even test the car's, you just expect that the tire pressure is correct, tank is full?
Expect that something "just" works is exactly why pilots have checklist's.
Expectations are the main point for disappointments, you would never do that with software right?
Hilarious you used this as an analogy since software development shops are notorious for cutting corners when it comes to QA.
You facetiously implied that every company fully tests software before it gets to production. Oh boy, do I have news for you...
Note the word "fully" as the variations of what gets tested is so broad, I don't even know where to start to explain this to you.
>Oh boy, do I have news for you...
Nah it's ok, just happy that i have colleges with a much better mindset and risk-management understanding.
And i stop here, since you try to change what i really wrote.
Not true the point was you pay for it (cloud), or you do it yourself (but then do it right, and not like a amateur who build's his first "gaming-pc").
And if you do it yourself you can still be very much competitive vs cloud.
Again, still avoiding the point, but oddly enough proving the point. You assume everyone isn't an amateur and knows how to build and maintain server hardware. Furthermore, because the market doesn't have enough talent to support all of the companies that exist, consolidating this to a few vendors who do have the expertise is what makes sense (economies of scale) and is what the market already decided.
Please read, that was my comment:
>>Not true the point was you pay for it (cloud), or you do it yourself
>You assume everyone isn't an amateur and knows how to build and maintain server hardware.
Yes that i assume, correct. Otherwise i would not call it "maintaining", is a amateur maintaining your car? Your software? If you have just amateur's handling your hardware it's probably better to pay a cloud-provider or pay a integrator todo that.
This is what I do. I rent bare metal from Hetzner and OVH. I also have some hosting hardware right at my place. It saves me a ton of money and no I do not spend any meaningful time to administer. All done by a couple of shell scripts. I can re-create fresh service from the backup on a clean rented machine in no time.
As for cloud - if I need to run some simulation once a month on some bazillion core computer then sure. Cloud makes much sense in this particular case. I am sure there are other cases that can be cost effective. Bot for the average business I believe cloud is a waste of resources and money.
I enjoyed server admin, "back in the day", when your servers were pets and not cattle. But of course we have to make tech just as expendable as our workers, business school demands it! What if your pet server gets hit by a digital bus?!
Of course, the newest idea is for creating and destroying machines automatically... that outside of the could is quite pointless but people want it anyway. I imagine seeing all that orchestration working must be even nicer than a machine autoconfiguring, but I am yet to see a place where it just works.
If I was a manager at FedEx I'd definitely spend some resources to DYI and send some of those millions over to my direction instead.
This isn’t really a meaningful analysis though. It’s just “when you do things in house there are things you have to do”.
It’s like saying, “I would rather retire than clean the toilets, restock the toilet paper, etc” in a discussion about whether to outsource your bathroom maintenance. Doesn’t tell you what’s cost effective.
The promise is to be able to pay up front for a rack that will function as a highly capable VM, storage, and/or compute host, without any of the overhead that Dell, HP, and IBM bring. Just plug it in and start giving it workloads to do. All config can be done through the web-based management console or via the API, just like AWS.
All of that can be handled by your colocation facility. In most cases you won't ever reach the scale where building your own DC makes sense.
> Then you still need to pay VMWare for a decent virtualization platform
Should still be cheaper than paying the AWS premium including for bandwidth, not to mention that you don't always need virtualization. If all you need the bare-metal for is a handful of machines to do a very specific task that's too expensive on AWS then running directly on the metal is an option (and leave on AWS the stuff that does require the convenience of virtualization).
> I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.
Agreed. Most companies shouldn't ever deal with hardware directly - just rent it from a provider and let them do the maintenance.
One of the visible signs of an unrecognized (and therefore, unresolved) dilemma is oscillation between two poles.
I got this from the late Eli Goldratt and the problem-solving tools he created. One of those tools is the "Evaporating Cloud," which is a way to visualize a dilemma of some kind.
I'd suggest that there is an unresolved dilemma here: Should we rent or own data centers?
Over a period of years or decades, you can watch these things flip-flop between two extremes.
It's interesting to me that this dilemma is kind of like a specialization (in OO terminology) of a more generic dilemma, which might be something like: "In general, do we want to own or rent the things we need to run our business?"
If your turn your head a bit, close your eyes a bit and squint, you can see how a dilemma like this can apply to things like "What should be our policy regarding employees vs. contractors? Should we try to hire and retain over a long period of time, or should we rent the people we need to get the job done and release them as soon as we are done with them?"
The overall point is that whenever you see flip-flopping between two poles, think "Unrecognized dilemma."
Sorry if this is vague.
If not for the rent seeking behavior of AWS and its cohort, the answer to ”own or rent” would be clear. The answer is “yes”.
Running a data center means you have your own sheep, which means you need shepherds. Shepherds are very useful to have when you have a question about sheep, especially when those questions are about how to manage or use sheep to best effect.
An approachable shepherd can save the rest of the company a lot of money on missteps and bad assumptions, but if you start getting rid of all the sheep, the shepherd will leave too.
We should be using cloud providers for DR, and for regional load balancing. But the company should be maintaining at least one data center of their own, in the same time zones as most of their developers.
I mostly blame Dell and IBM for this. IBM experimented with making server rooms easier to maintain 15-20 years ago and didn’t make it stick. Others ran with some of those ideas. Dell… I don’t know what Dell has done but I know nobody has been writing about it, so from a visibility standpoint they have done nothing.
If/when someone makes it easier (reduced labor) to manage your own servers, the pendulum will swing back.
It wouldn't take the most creative exec in the world to make a plausible looking case for changing in either direction even based on the same numbers. A bit of wishful thinking about which costs are included or excluded is probably more than enough.
Most of them were renting most of the infra anyway:
- Co-location (physical space rented)
- Managed service to keep the lights on (power, back-up generators)
- Leased hardware that gets replaced every 3 years
- Managed service for switch, firewall, etc. monitoring
- ISP, back-up generators, etc.
> but that will likely be many, many years down the road
They're going to explain to their future bosses that buying land, building a huge building, buying servers, figuring out cooling, setting up redundant ISPs, etc. etc. is somehow going to be smarter? Explain that one to me?
If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space.
There's a justifiable niche where you're too small to justify running your own server/below the need of one full-time IT staff where the cloud makes sense, and startups temporarily benefit from the ability to rapidly scale on cloud platforms. But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins. And they're still going to end up paying their own IT staff to handle the cloud management in addition to (by proxy) paying the cloud provider's IT staff to manage the hardware.
So, then why even provide the distinction of rent vs own if thats the case?
> you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc.
And you're also sharing the costs with other people. Clearly FedEx is saving $400M/year while also funding Bezo's trips to space, no?
> But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins.
This is literally not the case without the details so stop speculating. Every F500 should (and probably is) be doing a rent vs own calculation and determining the TCO, and then making the decision from there. It's not unilaterally the case, it has to be assessed.
I wonder how much they'd save if they rebuilt their infrastructure with own DCs. There are huge savings in decommissioning mainframes, switching to free software and automating more. At the scale of Fedex, I don't think they'll spend much more on hardware and ops than AWS.
The cloud just offers an additional middleman that also gets a profit margin.
You get your support in one place instead of Hodge pogge of ... yeah we rent server from X but software that runs on it is from Y and in reality provider Z is support so now C has to agree for physical access to the server and now align stars to get all 4 working together instead of shifting blame around to fix anything.
And you hope the Amazon employee makes decisions that are favorable to your business, even though they do not care about it.
The problem with building out a new state-of-the-art datacenter - or several of them - is the enormous capital expenditure you've just put on your company's books, not to mention the operating expenditure of all the people that will be required to run it. Yes, it's true that as time goes on, you can claim some tax advantages in the form of depreciation, etc. on some components of this new huge outlay, but at the end of the day, when the company misses quarterly or yearly expectations from Wall Street and the stock price takes a 10% hit, the "blame" gets squarely laid at your feet - you are, after all, "the problem". You spent a shitload of money provisioning future resources for the company's (expected) growth.
Meanwhile, in Cloud Cuckoo-Cuckoo Land, your rival has migrated all or nearly all existing infrastructure to the cloud, thus including a significant op-ex, but not nearly as large as the enormous cap-ex you've just incurred. They're hailed as a hero. A goddamned visionary! Look at all that money they're going to save the company! Nevermind the fact that they explicitly instructed the IT department's head to provision only the necessary resources for current operations, after all, the "promise" of the cloud is that you can just spin up whatever you need in a few minutes, anyway. And besides, the cloud deployment of all the company's servers and the necessary expansion won't incur a significant turning of heads until long after our visionary executive has jumped ship to another company for more pay, a corner office, and a better stock compensation package. Best of all, he say that he saved the company $XX millions of dollars over your plan, and it can be said legitimately, even though it is, of course, inaccurate.
If you're huge, it's never cheaper to farm out the administration of your critical infrastructure to qualified experts. But because of the dominance of the Quarterly Report Cycle in modern business, this gets swept off by the wayside as "outdated thinking".
I say this, by the way, as an IT professional building out cloud solutions for companies. For a lot of small-to-medium sized businesses, and startups especially, the cloud makes sense. If someone at Federal Express thinks the cloud makes sense, they're looking out for themselves, not the company.
Big companies need this too. Some team wants to spin up some new service to try out some idea? In cloud-land they push a button. In "rack & stack" land they have to wait for Ops to purchase the hardware and provision it.
Cloud makes it cheap and easy to try out new stuff.
All transactions, no matter if in the cloud or on prem, go towards another company’s profit margin.
It's really just like the utilities these days, 98% people will not dig their own well and install some sewage system and buy a generator, but big companies can still opt to have them on-premise installed, even though some of those are just backup systems.
there is no return for the cloud for 98% of us.
You mean there are fewer companies making large investments than smaller ones?
As someone who's been building an off-grid solar system this is largely the case. I've spent thousands on panels and batteries, my electricity bill is only a few hundred a month but we have frequent outages during storm season. It's definitely more of a feel good and control thing. If I consider the cost of running the electricity to our remote location, I'd probably barely break even in 10 years if I'm lucky.
I worked in a building during rolling blackouts in California. The Tesla Powerwall on the building was "screaming" (ultrasonic harmonics from piezo components) 24/7. The security guy actually came to get me to ask if the thing was going to explode.
Clearly, the building owner was load shifting and making quite a bit of money from it.
They shouldn't, though. If powerwalls are that great, it should be more cost effective to install giant banks of them in cheap warehouses outside major metros since you'll have economies of scale on installation and maintenance. Utilities are sorta getting into this game, but it's more common at the individual level, despite being more expensive.
This works only if the electrical grid has the ability to consume back your stored power. Most electrical grids in the US would have problems doing that.
For example, UCSD has their own generating facility; however, SDG&E was sufficiently backward that UCSD was unwilling to go through the grief necessary to put energy back into the grid. Therefore, UCSD only does load shifting or disconnects the campus from the grid during rolling blackouts.
Because of the poor energy grid transport, storage batteries make the most sense when you can consume the stored energy locally to minimize your grid consumption during high energy prices--ie. you have a high-rise office with lots of air conditioning or a manufacturing facility.
Luckily, it does work for most of the US, especially desert/arid areas.
> but I see fewer companies building collectors in places with cheap land and ample sun.
They are plentiful in Turkey and some southern EU countries like Greece/Croatia, I believe. Not sure about deployment in the US.
Generating solar for local use saves the cost of transport.
High-voltage line losses can be as low as 3% per 1,000 km. That should be easily offset by the better location and better angle/tracking.
(The wise thing to do to mitigate financial impact may actually be starting the reintegration process right now.)
Of course they should make some effort to ensure there is competition. That means they are careful to ensure more than one provider exists even if it means taking a slightly higher priced option. Also contracts end early if the company is bought out. (that is if AWS, buys google cloud - see above about ensuring there are competitors in the market)
My guess is, we'll rather see some consolidation, like in every other segment. Which also means considerable expenses for the remaining players, who will experience increasing need to regain some by pricing. As theses players are sharing roughly the same boat, this will show quite naturally some characteristics of an oligopoly. At the same time, self-managed infrastructure will have become a rarity and costs of reintegration are prohibitive, which should allow for some elasticity in market prices. Also, any investments towards reintegration won't show results during the turn of current management, minimizing chances for this to happen, yet again. (This is not a level playfield anymore.)
Sometimes tech debt can be used instead of financial debt!
Either way, I think you're spot on with the "training" part. FedEx was one of the first companies to raise significant capital and pursue a business dependent on technology. It also treated its people very well so relatively few left since the 70's. I think they just didn't train the next generation enough and they realized that those skills are disappearing rapidly because no college grads want to learn old & boring stuff and everyone who does know it costs a lot to keep (FedEx still heavily relied on COBOL when I left ~5yrs ago).
How costly is it to maintain that level of talent? Do you think top server admin talent is screaming to work at FedEx or Amazon?
With a good design, there is always that implicit threat that they could move back to on-prem with little effort.
There is no grand plan, there's just evolution.
One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down.
I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.
Now, they might find that software is a differentiator for them, but that's different.
HNers need to get their heads around this: the economies of the could represent a fundamental, secular shift.
Imagine if Fedex designed and made their own delivery vehicles. Then they 'outsourced' that to Ford/GM. You might say 'look at Ford's amazing margins, look at the money being left on the table' - but in reality, it's still more cost effective to 'buy vehicles' the to 'DIY' them.
In the 'long run' those Azure/AWS margins will erode - and/or - the long tail will creep up on them. Basically data centers offering less, for less, and companies realizing they don't need the complexity of AWS do do basic things.
What that will look like is hard to say.
With hardware it meant 'pivot to Asia'.
Will this happen again? Chinese companies creating large data centres in the US with minimal staffing, monitored and operated out of China?
If we were not in a giant geostrategic kerfluffle with them, then yes, I would say that is the future. But given security issues etc. it's probably not.
Can India do that? Maybe.
By contrast, generally speaking, cloud providers are in an excellent position to use price discrimination to extract the entire surplus value of every transaction. And the flexibility and reliability provided by FedEx will to a significant extent simply be that of their computers.
(As chrisseaton pointed out, Boeing is in a cloudier position.)
Especially to the extent that special services are not leveraged, then there are cheaper alternatives. A lot of IT is just 'instances' not 'fancy cloud queues'.
AWS pricing is very transparent and people can make the decisions they want. Generally speaking, AWS does not price discriminate on the basis of 'business model' and their margins are meaningless with respect to the overall business efforts of a Big Co.
I mean - unless you are doing big AI crunching, or 'free hosting content' - then frankly the AWS bill is not going to be a huge line item relative to overall operating costs.
And this is where people in my age range (20-30) will get to swoop in if we invest the next 10-20 years learning about and tinkering with server hardware.
I highly recommend it for everyone.
"Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?"
from the post reply, in case it's not available from the URL:
2 days ago
· edited 2 days ago
Gold2Eureka!Bravo!Today I Learned
Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal.
One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired.
So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually.
In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge.
This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant.
The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm.
The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow.
As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run.
Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha
For example FedEx flies Boeing jets. It's completely reliant on Boeing for parts for those jets. Presumably Boeing can charge whatever it wants.
Your company is always going to be reliant on other suppliers and contracts. Unless you're planning on building your own independent country in some location that has all the natural resources you need.
Nobody said that FedEx should build their own server hardware. And that's what you are comparing it too.
FedEx could just buy server appliances from e.g. Dell (buying the Boeing jet) and operating it. Because paying some other air cargo company will eat a lot of their margin, the same with Cloud infrastructure. They are not a startup which could better invest their time in development, they can without any problems hire some people to administer their server fleet. When they switch to a cloud they will likewise hire some engineers only managing e.g. AWS to administer it.
Your negotiating power is then in the momentum of change. If GCP is working better for you than AWS right now, send your new spend to GCP where possible and where stuff is rotating out, move the new stuff to GCP. If you just got a good deal from Azure, start moving in that direction.
This is one of the 'big company vs small company' things, by the way - it doesn't make as much sense for a 100 person startup. But FedEx are a 300k employee juggernaut who spend $75B each year servicing $100B of revenue. They'll have a lot of tech in use.
Anything in between will just mix all the problems of a cloud with all the problems of running a datacenter.
I believe the self hosted options generally offer vastly more flexibility and can innovate at a faster pace. But only FedEx will know if it is a good decision, perhaps their infrastructure was indeed not competitive. But I heavily doubt they will save 400m in the long run.
AWS support is actually awesome and you usually get responsive within a day, even smaller businesses. But cloud hosting is only useful for certain scenarios. In this case it is Azure and I believe Microsoft currently innovates far slower than Amazon. Of course Amazon could be a direct competitor to FedEx in a few years too given they do more an more logistics. If they aren't a competitor already. In this context I would understand the decision to close their data centers because it is hard to compete with Amazon in the cloud space... a computing alliance with Microsoft would make sense.
This doesn't pass the smell test at all. Datacenter ops are way more complex than AWS ops; why would someone on cloud need IT support for VMWare, BMC Ops, Power management and generation, supply chain management and land leasing/renting? These are the few roles, off the top of my head, that just go away when moving to the cloud at FedEx's scale.
But sure, at the scale of FedEx this is true, they also probably have a very high percentage of administration only. Although I still think the cooperation with MS was still a strategically planned after paying attention to Amazon.
FedEx might have a viable alternative to Boeing for repair parts, but not for planes, and more broadly I think you could make that argument about US airlines in general in the late 20th century: despite things like the MD-80 (before the merger) they really had no alternative to Boeing. It is perhaps not a coincidence that the net profits of US airlines in the late 20th century were almost exactly zero, while Boeing was and is one of the most profitable companies in the world. So far Boeing isn't squeezing FedEx that way, but not because it can't.
(I do note, though, that FedEx owns its own jets rather than renting them.)
But I think there's a different sense in which informatics are more core to FedEx than even flying. FedEx is a corporation, which in its essence is a set of business processes: relationships, practices, and information; unlike a 19th-century railroad, its physical assets like airplanes are relatively expendable by comparison. Historically, those processes happened mostly in people's heads, especially the heads of its managers, but today the vast majority of FedEx's business processes are automated, which means they happen in FedEx's data centers.
And whether those automated business processes happen at all, and how well they happen, is a matter of competence in informatics. Dan Luu argues convincingly that Twitter's kernel team has been key to their ability to execute in https://danluu.com/in-house/; FedEx needs not only data centers but a kernel team and a convex optimization algorithms team.
It's extremely common for companies that are deeply dependent on a single supplier or customer to end up either merged into that other company or bankrupted by the situation.
As for the country, when I set it up I'll be needing yeomanry regiments. You in?
Edit: removed rude phrasing.
No that’s not what I wrote.
You don’t buy a jet outright and forget about the supplier - you’re eternally reliant on their service, certification, parts, etc, as regulation is so high.
That may be your comparison but it’s not the one I meant.
And believe me, I know a thing or two about being discourteous.
In fact, FedEx likes to buy out Boeing facilities so that they're not "completely reliant" on Boeing for anything.
As for the more general cloud v. on-prem debate, I usually don't like analogies, but the plane analogy is good because jets and logistics-management systems are two things that FedEx needs to handle extremely well to compete.
So, what does FedEx actually do wrt jets?
FedEx both owns and leases them. It mostly owns them outright. Its owned jets are a mix of new-ish to very old (think DC-9s). It also leases and some financing magic (such as some big lease agreements for new 777s a couple years ago).
The reason is that FedEx generally gets excess value from owning rather than leasing, but there are some circumstances where leasing makes sense.
Same goes for "cloud" deployments. The correct answers to cloud "versus" on prem for large organizations like FedEx - "it depends," "that's a false dichotomy," and "it's almost certainly a mix of both" - are neither interesting nor simple, and so those answers don't get execs' attention...or headlines.
For FedEx to brag about taking an extreme position on cloud deployment is breathtakingly foolish. If I were an investor, I'd want to hear something like, "Based on a careful analysis, we've decided to shift certain specific operations to cloud-based systems. We plan to maintain control over the infrastructure and operations of mission-critical systems that have demonstrated resilience." (I'm assuming the latter are systems like the financial institutions rely on, where purpose-built stuff like mainframes using IMS have demonstrated near-zero downtime and darn-near-bug-free software for decades, because FedEx, like banks, needs to handle lots of simultaneous transactions in real time without error or deadlock.)
 Example: https://www.ch-aviation.com/portal/news/102874-fedex-to-take...
This, in particular, is a bad argument since jets are replaceable and interchangeable.
Possibly, the only thing that is not replaceable in FedEx is their information system.
This isn't a one size fits all thing. For a company like FedEx I don't see the advantage of owning their own data centers. They just don't have the scaling challenges that would require that. Data centers or information infrastructure as you put it does't fall within the core competency of FedEx, so I don't think it's a fair comparison to say its like outsourcing upper management. I think its more fair to so that FedEx building their own datacenters is like building their own roads to deliver packages on.
For a company like Facebook or Google, the difference is clear. They need to handle such a high scale of traffic and volume of data that they need custom infrastructure to be able to scale efficiently at significantly reduced costs. The same reason it made sense for them to invest in building their own databases, because the existing options didn't meet the scaling requirements. FedEx won't realistically be needing their own database or infrastructure any time soon.
I don’t know about that. They are a global logistics organization that rivals Amazon in scale. They are not parsing web scale data like Google, but as far as meatspace data goes they are they seem to be about as big as it gets?
Your metaphor is a little bit off. This is more comparable to FedEx renting vs owning their fleet vehicles.
Obviously they aren't going to be setting up foundries to to scratch-build engines and network switches. But it probably makes as much sense for them to own the buildings & computers running their logistics software as it does for them to own the buildings and vehicles running their logistics hardware.
As an aside, I'm sure FedEx would LOVE to get customers locked in to private FedEx roads with private FedEx addresses that no one else is allowed to deliver too. Thankfully, our system of public infrastructure is robust enough to make this infeasible.
In the United States (along with Morningstar Air Express in Canada), FedEx Express operates FedEx Feeder (and for Morningstar, mainline FedEx service) on a dry lease program where the contractor will lease the aircraft from the FedEx fleet and provide a crew to operate the aircraft solely for FedEx
Scaling is one of the big cloud advantages. Static Known workloads will almost always be cheaper on Prem than dynamic workloads that need automated scaling
One of the big ways an organization can save money is if they can scale up and down their loads on demand
But if you have a static 24/7 workload almost universally I can build a onprem solution that is be 50% or more cheaper than cloud
Depends how seriously they're locked onto the postman/route inspection problem. Which is a factor for delivery companies and, depending on how hard you chase it, a driver for almost infinite compute. Once you're playing that game, the economics look very different to most 'not an IT company' IT needs.
That is really not true at least in the USA. There are all kinds of taxes and fees for using the roads with large trucks. Have you ever seen the weigh stations on the sides of the highway in the US? Those are there to fine the trucks for what they are carrying if it is beyond what is allowed among other things. You said km though so you probably are not in the US.
There's also tax built in to every gallon of fuel.
There are also flat annual registration fees you have to pay as well.
In this case, if a cloud provider gets too monopolistic then the BATNA for fedex is to re-hire (or train) enough sysadmins to run their own systems again. That puts an upper limit to how much a cloud provider can charge for their services, in addition the competition between Azure/AWS/GCP will also keep prices somewhat down. The cloud providers know this and will price accordingly.
I don't think FedEx re-hiring a competent sysadmin team and rebuilding data centers once they've lost that knowledge is merely a question of spending enough money on the problem; it's the kind of high-risk IT project that often sinks companies.
Uh yes, exactly?
Clouds give small customers the first hit free as a loss leader. With very large customers, clouds have to be price competitive because if you're at a large scale, it is totally worth spending millions of dollars for a 5 year plan to change clouds. The people who get screwed are the medium to large customers for whom the cost of changing clouds is too high to recoup in a reasonable timeframe. The moral of the story is be very small or very large, but avoid being medium sized.
You've literally just explained why a large company would be perfectly willing to be overcharged by a cloud provider in order of millions of dollars. They would have to pay that for migration anyway and there is always a risk of creating disruptions in the process.
It's a huge project but there's not a lot of actual redesigning.
It’s also not just the cloud, enterprises know they get the best pricing when they commit to a vendor and the vendor also bends over backwards to accommodate these large companies. As long as there are 2-3 cloud providers big enterprises will be just fine.
Once your start up has become an established company and are so busy that you are no longer able to scale down regularly, it seems that's the time to start having your own compute vs continuing to line the pockets of the cloud providers.
This makes me think computer infra could potentially become a commodity.
Most cloud providers do let you run your own software on a managed substrate of infrastructure. But they also let you, nay, encourage you to build on top of their abstractions instead.
This "ultimate" endgame has been predicted since at least 2006 and we've yet to see anything but price decreases on cloud services. Tons of labor has been invested into deliberate cloud agnosticism with no apparent results. I am fully on board with economic arguments that favor data centers over cloud for some organizations. I don't think the capture-and-increase argument holds evidential water or ever will in a competitive environment, and the environment is more competitive than ever.
The cloud market is so large ($ TAM; also growing rapidly) I think there will always be a tailwind of investment and innovation that prevents monopolistic stagnation.
vendor relationships do not have to be hostile. AWS has been around for a while and has never raised prices, some services have gotten 99% cheaper.
yes there's some hypothetical day where they say "tricked you, all that stuff about margin being opportunity was a lie" and they try to extract value short term while burning their reputation
but every day that goes by where this doesn't happen is $$$ wasted on trying to avoid it
I'm betting it is not this kind of thinking exactly that causes the problem.
(Oh, and do take a look at the vertical consolidation of the Japanese industry.)
This is silly. There is a lot more to FedEx than hosting physical compute infrastructure. FedEx doesn’t own all its own airports. A CSP is just another vendor.
That's a competitive environment.
How many major cloud providers exist, and what could they do to make it difficult for FedEx to leaving for another one?
Plenty! AWS is HUGE but hardly a monopoly.
>what could they do to make it difficult for FedEx to leaving for another one
If the guys running this at FedEx are smart, not much. There are ways to deploy all of this in a platform-agnostic way.
I think the digital transformation to watch is the US Government's IRS Tax Cloud. Never been into taxation law & policy wonkery myself, but Tax Cloud is purported to Save Trillions in TCO!
> Clown Computing is an elegant, distributed, breach of trust.
> Take customers, as many as you like, convince them to hand you all their data,
the management of said data, the authentication and, while you are at it, the
DNS. Oh, and they pay for it too!
> If this was done in real life it would be called a robbery, possibly at gunpoint, definitely in broad daylight. It used to be the case that Oracle licensing was deemed the pinnacle of IT robbery ('90s) but it looks positively quaint now.
But by the time this becomes a problem the executive team will have cash out nice bonuses from the short term gains from cost savings...
Apple fairly notoriously hosts iCloud in other company’s clouds, and they’re a computer hardware and software company! If you’re less sophisticated than Apple, it’s a no-brainer to go with clouds.
They’re in the datacenter game long term. In no universe does Apple “host iCloud in other company’s clouds”. Apple is notoriously a control freak and would never place their strategic services at the whim of cloud AZs. That idea was proposed and quickly eliminated. (Believe it or not, it is possible to spend that much on Azure/GCP and still only use it for basically blobs. How do you think they moved so easily
My argument is not that specialization in informatics operations is unprofitable; rather, it's that specialization in informatics operations is indispensable for maintaining the negotiating position required to earn a profit. I'm not arguing that there isn't room in the system for the cloud operator to earn a profit; I'm arguing that after committing themselves completely and irrevocably to cloud operators, FedEx will no longer be allowed to earn a profit, or rather, just enough profit to keep their stock price from collapsing.
Now, it may be that the advantage of specializing in informatics is so powerful that this situation is unavoidable, and the only viable option for companies like FedEx is to become, in a sense, franchises of Amazon, Microsoft, Google, or Baidu. You remember when this happened to companies like Intergraph in the 01990s: they were reduced to undifferentiated Microsoft resellers. And it's not clear they really had another option. I'm not confident that the situation is currently so extreme, but maybe it is.
As other commenters have noted, you were pretty comprehensively wrong about iCloud, but that's a minor supporting point.
Is the expectation data centre operators will see margin expansion? (Genuine question.) I had thought data centres were increasingly becoming commoditised.
Balancing vendor lock-in against efficiency is the remit of a half-competent CIO or equivalent. Going cloud doesn’t require using every niche AWS feature.
Data center/SaaS is somewhat naturally monopolistic due to cost of switching though. Legislation should target "high cost of switching" areas to make them more competitive. Translates to lower prices, better for society in the end
That is, all companies ever.
The opposite of what you're proposing is "do your core business".
Why would fedex be better at running computers than Azure is?
Why does FedEx use vans built by a car company? They should build their own vans. The tires should clearly be made from rubber that FedEx vulcanized themselves.
I've seen huge insurance companies outsource not just their hardware, but their ops too.
McDonalds may be in the real estate business (and not the burger business), but FedEx is not.
The cloud is the only reasonable way to keep up with exploding compute demands.
The question is how much of the savings end up in whose pockets, or whether the price at which they actually sell exceeds the TCO of running it yourself.
Outsourcing at least the commodity part (dumb hardware) seems reasonable, and migrating to a competitor doesn't seem like too bad of a BANTA if you aren't locked in.
If they're building it on AWS only kit and locking in, then they open themselves to risk. If they build it on standard VMs running on AWS, Azure, Digital Ocean, then they can shop around at contract renewal time, just like they can with Mercedes vs Ford vs VW trucks
As long as they stick to services that have overlap been providers (postgres, mongodb, nfs/cifs, etc) or transformers (s3, azure blob storage) and package their code in somewhat agnostic format (docker images, wars, what have you) they'll be fine.
Big organizations often do things that seem wasteful as a way of dumping organizational cruft.
I agree with you, that the hype train "cloud = total freedom" (whatever that means) is rubbish. More and more it seems, that a cloud model is just yet another option between Mainframe, AWS, Google Cloud etc.
Far as I can see the only arguments here are that either cloud compute is naturally monopolistic, which seems a stretch, or there's another factor that will lead to ruin here (e.g. long-term connectivity loss between your and your compute in a particularly nasty disaster scenario).
Like, I don't care* if my baker's goals are orthogonal to my own. They just provide crullers for the morning stand up and I can get them from other places despite the insurrection that would occur if staff ever went without. I can likewise not care if my cloud provider's goals are not aligned, because if they screw me I can just shop at another bakery, so speak.
All of this said, outsource to cloud isn't free. You have to have an exit, or at least transition, plan. You have to architect into the configuration, not just transpose what you have. You have to maintain platform-agnostic restorable backups of the information that's critical to your organization, and have an operable proven plan to bring in online. Failure to do these things, which I think is the case for a scary number of organizations that have 'gone cloud', could indeed lead to ruin.
I know this comment is weird. I'm in an odd space here of agreeing with your conclusion but mostly disagreeing with the premises on which you reached it.
*until we see a cloud provider that also owns major fab i.e. if Intel or AMD went into direct cloud provision and took it seriously for about a decade I'm pretty sure it's game and over the world itself would implode from the sheer monopoly that would ensue. Until then, however...
As long as they have backups, yep, that's a quite possible alternative. And backups cost very little, and don't need to run flawlessly 24x7.
They will probably pay through their nose either way (backups or not, migrating or not), because that's how the cloud works. But well, that's for their bean-counters to count. As long as they have backups, it's not strategy defining or an existential risk.
(Anyway, nobody goes around talking about BATNA of the proprietary, rented mainframes. I wonder why.)
That's correct assuming monopolistic behavior. I believe pricing in information infra will be a race to the bottom. We've not seen a lack of competition in this space.
But there will surely be a backlash and an - at least partial - return to operating your own datacenter for some companies in the future.
This is permanent secular shift.
The advantages of the cloud are just starting to be realized.
That said, for some kinds of operations, they may want a lower cost cloud provider with a more basic stack.
Secondary operators of the world need to get together and start offering a standard stack for basic things so that people can wean off of Azure and AWS.
Over time competition between the 3 major clouds combined with hardware getting cheaper will mean there is not going to be a massive jump in prices, if any at all.
It is not impossible to design relatively cloud-portable server software.
Also I think FedEx, and companies in general, would prefer to increase revenue rather than reduce costs (by running their own data center) in order to increase their net profit.
If AWS et all get too greedy, competitors will pop up saying 'your margin is my opportunity' and market competition happens for something that is a commodity, and becoming more so as time goes on.
Cloud setups are useful, as are many ways of their associated architecture/infrastructure elements. Saying no to these is problematic too.
It is true that cloud computing has become a monopolistic, categorically non-neutral sector. They'll have you over a barrell.
So, dilemmas. Difficult strategic choices.
I think the reality in this case is that the cloud providers are little different than the incumbent (ie IBM or some other dead platform). Mainframe is a sole source solution, and IBM is always trying to extract their own pound of flesh as they manage that business down to zero.
At scale, you’re better off managing a handful of suppliers (probably Microsoft, Oracle, Azure, IBM) than dealing with one, and figuring out how to sustain the business as the workforce literally dies off. Nobody with half a brain is getting mainframe skills.
I’m not a fan of FedEx, but they seem to be a company well aware of its limitations and able to take action to correct. For example, they went to “outsourcing” the last mile delivery to independent contractors first when they figured out (almost too late) that e-commerce needed ground shipping. They sort of did Uber, except they didn’t have the ability to just break the law.
they can rent hardware only, and still run their own infra (kubernets and all stuff on top on it), then they can negotiate reasonable prices and migrate away if they want to.
Cloud services are about to evolve in ways that non tech-specialized companies can replicate on their own. Cybersecurity requirements will grow and bandwidths will be pushed to the limits. Just leave it to the professionals.
You say that like it's bad.
The optimal amount of cloud services for an established company like FedEx is 100%, not merely with a "disaster recovery plan" but with live, 99.99% redundancy by which I mean two almost exact systems running nearly simultaneously (within a second or two of one another) on two completely different networks.
FedEx enjoys almost all of its competitive advantage from its physical network. IT is not core to its business.
Here's the problem... almost no company actually does disaster recovery and "parallel redundancy" properly because most C-level executives only pay lip service to it.
Therefore, the whole notion that disaster recovery and "parallel redundancy" don't work is predicated on the false notion that companies actually have proper disaster recovery and "parallel redundancy" in the first place.
If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point where this is no longer true, and even huge corporations can save money in moving away from their own premises?
Take FedEx, their operations are highly dynamic - per day but also per period. The number of packages sent, and being transported, vary greatly between a random Tuesday 15:00 and the weeks before holidays.
In such a scenario, with your own infrastructure, you need to overprovision, by a lot, to be able to handle the heaviest load possible, and then some margin on top. And that capacity is wasted what, 90% of the year?
Meanwhile if you subcontract that part on AWS, it's their problem, and they can afford to handle it, and you only pay for what you actually use when you use it.
Taxis are "only pay for what you use", but owning your own car is often cheaper (even if you only use it, what, 1/12 of the day?) and more convenient.
Exactly, so as long as it's not you, you should be fine.
Let AWS figure out how they wish to do it for you.
It's rare, but there are enough strange instance types that it's possible.
Taxis are not scalable, you can't make one human drive 100 taxis. Meanwhile you can add entire new data centers with barely any extra burden on the software teams that manage them.
We own our own car out of convenience not cost savings.
The same routing algorithms have to run either they have one package or 1 million packages.
Plus you need to keep the history of the operations for months if not years (meaning that you will store both holiday and random Tuesday data anyway).
So where is this flexible cost exactly?
I could easily see FedEx can save money by dynamically scaling capacity to track the arrival of their planes or trucks.
So for practical applications they are constant time.
So they should be able to handle all of this on a raspberry pi, if the size of the input has no effect on processing time.
It is not difficult to find a solution to routing problems, the difficult part is to get probably near optimal solution (<1% optimality gap).
The issue is that there are exponentially many of them and you cannot easily rule out that there is no better solution than the one you have on hand.
That is why we solve these problems for as long as possible with as many resources as possible.
Certainly you know that running a routing algorithm 1 million times because you need to route 1 million packages is going to need 1 million times more CPU time than one package, right?
The number of variables is not a predictor of time and average complexity for integer programming.
We can solve some problems with millions of binaries in minutes. There are other problems with a couple of hundred of binaries that we cannot absolutely solve to optimality.
Sorry that this is your typical sorting problem.
If it takes 1 second of CPU time to process the path for 1 package, then it will take 1 million seconds of CPU time to path find 1 million packages. You can add more CPUs to run multiple path finds in parallel (ie, use 4 CPUs to cut the wall time to 250,000 seconds), but there's still 1 million core-seconds being spent.
Perhaps there isn't the mindset/ability to actually execute this though.
This works great for everyone, the hyperscalars have much lower costs than even the biggest enterprise customers so the company get a good deal. And the cloud company makes some revenue off the minimum spend (making capacity planning a lot easier) and they know once the compute and storage there it will be very likely the company starts using extra managed services.
The dollars-and-cents are details. Those matter, of course, but don't base your entire analysis there. Staying vertically integrated may help fedex be 4.713% more profitable today, but might miss out on important growth areas and become a dinosaur. Or maybe the important growth areas involve data center innovations, and staying vertically integrated allows them to exploit those opportunities. So it's not always an easy decision, but it involves more than the present-day costs.
Ultimately, this is an opex vs. capex decision. Data centers are expensive, and require a lot of up front capital to go into the ground. FedEx is worldwide and moving more technology to the edge, so buying land, designing and constructing buildings, and then operating them for 10 years or more in a large number of locales requires a huge investment. They can take that money now and solve their problems immediately.
They are also not saying what they mean by "closing its data centers". FedEx announced a 10-year partnership with Switch last year to build and operate edge facilities. FedEx may be moving out of data centers it operates on its own into facilities built and run by others, and using "cloud-native" designs deployed on hardware in those facilities.
I don't think this sentence has meaning
The Mainframe is a sort of cloud, each one has usually double the spare capacity and you call IBM to unlock it.
The cost of running a Mainframe is measured in terms of MIPS based on amount of compute used and there's compute reserved for offloading things like JITs (zAAPS). Essentially a version of cloud compute costs.
I can understand if you're locked into COBOL with EBCDIC and can't find talent, but Z runs freaking Ubuntu now! I know IBM works furiously on modern ecosystem support.
Not shill for IBM, but what exactly is it that they're expecting to be so different from their cloud migrations? Or is everyone here caught up in the "Mainframe old" falsehood.
Here's the thing: A mainframe may have spare capacity, but you still have to call IBM to unlock it. In order to make a Z run in a cost-effective manner, you need to run at 90%+ utilization at all times - which is excellent for batch jobs that can be scheduled, but is difficult to achieve with on-demand loads.
You're paying based on compute available, not just compute used (unless I'm misremembering our contracts back in the day). Sure, the amount of available capacity can be changed, but a phone call to big blue is not automated. Cloud autoscaling is.
You're right that IBM mainframes are far more modern and cost effective than we assume. Sadly, they are still behind the curve of cloud hosting.
Respectfully, the hell you say. Cloud 'autoscaling' is something that has to be tenderly maintained by software engineers, who expect to be paid salaries and benefits and so on. It's not like FedEx can just rsync their data into the cloud and have all their software run forever. Instead, they need the engineering team that manages their existing data workflows, and now they also need cloud engineers to translate that into something that won't bankrupt the company, since they're moving from the mainframe world (where you keep your system loaded to an efficient price point) to the cloud world (where you are billed by the second).
Moving your compute spend from capex to opex is a perfectly valid move, but pretending the reason is that someone has to make a phone call once in a while is kind of bizarre, when the alternative is having to hire a whole new cadre of techincal talent.
I agree the occasional phone call removed from your workflow isn't reason enough to justify a giant cloud migration... but it is one of many reasons.
Ever work for a company with their own datacenters that were NOT cutting edge? The work to provision resources can be so painful that it severely inhibits prototyping. Being able to spin up resources in seconds with some API calls is very valuable.
It's probably this. I've spent time trying to advocate for the benefits of these systems, but its like shooting a super soaker into the sun.
I get my pick of a billion different hardware vendors, they’re all interchangeable and interoperable, every problem I can possibly dream up has been solved a hundred times before. The skills are commonplace and transferable so hiring doesn’t mean convincing some poor desperate grad to get trained in the company script, hires will actually want the skills, and you can find senior people.
A good example of where these systems come into their own is payment/ACH/wire processing. The consequences of these networks going down are so severe that it is worth it to construct an entire facility with the computer systems and business in mind from the very beginning.
Today, mainframes are more for the type of business that are finding the need to pour the literal foundations for their own datacenters. If you are even considering cloud as a viable path, then this kind of stuff is certainly not for you.
You're paying a premium for both the hardware and the lack of develops knowing EBSIDIC, JCL, Cobol, etc..
Actually, from what I remember of Oracle, that might be very similar. I remember having to pay license fees per core.
Mainframes are still king when it comes to transaction processing but having such a closed off ecosystem has screwed them.
Even FPGA's are more accessible to learn. On AWS you can spin up eg; an F1 instance and get a dev environment.
you don't need to call AWS. Negotiate a blanket deal and do whatever you need. Run whatever you want. Unsure? Check billing page.
If you go to the cloud you don't need to fire anyone for choosing IBM, you're not getting strangled by any Oracle contracts, you're not gonna lose all your data because of security holes in your use of Microsoft products.
You're also not going to be dealing with downtime because you couldn't find staff talented off to properly configure your Cisco networking equipment.
Your system administrators department is not gonna block any innovative employee initiatives because of the strain maintaining more projects puts on a deployment stack carrying 150 different projects but which was architected to solve a single business goal 15 years ago.
Imagine being a 67 yr old manager who just wants to be in the business of getting letters printed on tree pulp from Bumfuck, Idaho delivered to Louisville, Alabama reasonably effectively. Knowing nothing about computers or information technology, imagine the insane stack of perfect decisions that have to be made to get the IT infrastructure of a company like FedEX running.
Not saying going to the cloud is the best decision. Not even saying it's the decision I would make. But it does sound very enticing to just have all of those problems go away by throwing a couple hundred million dollars a year at Google, Microsoft or Amazon (btw lol if they go with Oracle or IBM instead).
>Your system administrators department is not gonna block any innovative employee initiatives because of the strain maintaining more projects puts on a deployment stack carrying 150 different projects but which was architected to solve a single business goal 15 years ago.
As someone in the “cloud” team at a legacy enterprise I strongly disagree with both of these points.
Cloud networking is as complicated as anything the Cisco people ever did but instead of CCNA you have certifications that barely scratch the surface of the complexity. So you get cloud people who barely understand the platform they’re administering flying by the seats of their pants. And instead of having a networking team to focus on networking the same people trying to figure out static routing across regions in AWS are also the people responsible for migrating EC2 instances from GP2 to GP3 but they were deployed with cloudformation which will replace the instance if your change of the disk type.
So getting to the second point this team will be totally overwhelmed and likely inexperienced so good luck getting them to do anything to help your “innovative” project because right hope they’re too busy trying to figure out how EKS is using up all the IP addresses in us-west-2.
Executives who think they’ll save on money by moving to the cloud are delusional. They’re also delusional if they think it’ll increase stability or resilience. And that’s not even getting into the EMR clusters the “data science” team spun up and left running at $30k/day.
I don't know why I have to explain this every time "data center vs cloud" discussions come up, but if you reduce risks, then you are in effect saving money.
I'm quite sure if you take managing your IT infrastructure as seriously as you take your core business, you can definitely save tons of money by handrolling your infrastructure.
There's also a big difference between doing so 10 years ago versus now, with all the enterprise grade open source solutions to infrastructure challenges.
Try telling a company like Catepillar who manufactures excavating tools to "take IT as seriously as you do making tools"?
> with all the enterprise grade open source solutions to infrastructure challenges
You mean like OpenStack? Have you ever been in a large IT org (non tech company...like a distributing/manufacturing company) that has tried to implement it? and then maintain it? Oof...
And I'm saying you're completely armchairing this analysis because you literally don't know any of these details.
> A company like caterpillar I think should be run almost entirely on no/low code platforms.
Are you suggesting a global manufacturer like Catepillar run it's global financial ledger on a no-code platform? Which implies building the code for it and then maintaining it?
> It seems you're trying to nail down an absolute
In fact, I'm not. The only absolute I'm trying to nail down is "you need to do a buy vs build, rent vs own assessment and make the decision there, neither one is unilaterally true without that assessment". Anyone trying to speculate about budgets in the $100M+ space is just heresay.
And exchange it with dependability risks
> you're not getting strangled by any Oracle contracts
Unless you go to the Oracle cloud
> Your system administrators department is not gonna block any innovative employee initiatives
They're still there, aren't they?
I'm sure they'll save many millions.
Earlier this year I left FedEx after 15 1/2 years of service. Every single day I used an IBM AS/400 terminal to interact with several systems to do my job, the bulk of my job was done via that terminal. Yeah, that's how old most of FedEx's tech is. A few months before I left they had just migrated one of our in-house systems over to Oracle likely as part of this. That said, the mission-critical system was having almost daily errors/downtime once migrated to Oracle soooo...
I imagine some of this is the simple fact that they need to replace these severely aging, no longer supported, IBM AS/400 servers throughout the company. They lost hardware support around 2020 and, if I'm not mistaken, haven't been made since like 2008 or something. That alone is going to save a pretty penny in new hardware and energy costs as well as free up physical space at often already crowded areas.
It'll also save time-lost costs. Any time power would go out at our building, the servers would usually be done for tens of minutes even with the generator kicking on. We'd lose a couple of hours a year usually to the servers that were in our building coming back online. A couple of hours, times 100~ employees at one site, equals a LOT of backlog being created which ripples through the company. While that few thousand dollars they're paying employees during that downtime isn't much, it would cascade and disrupt the freight handling. If a single package didn't clear customs in time, then it might end up as an overage and have to go to a bonded cage, that's now 2 extra movements added, that's freight planning for possibly multiple trucks that will be part of the delivery once the package landed in the United States, you might see a thousand or more shipments (that may or may not be single package shipments) now needing to be handled extra at a half dozen ports, even more sort facilities, and even more local facilities. That's just from 1 office losing power for say a half hour.
Moving those servers to a cloud provider should provide a much better uptime which should translate to a notable savings in the above situations.
The hideous frontend of greenscreen RPG programs is optional, you could replace them with Java if anyone cared enough about UX for internal tools (they don't).
The hard part with that stack is getting RPG devs - most of them are 50+ and expensive.
Now they'll need how many engineers to move the code stack (that probably no one knows) to some groovy stack with 12 layers.
But also, mainframes are extremely expensive, so they can probably do better on commodity hardware. The cloud is a way to rent a lot of commodity hardware.
The big-brained move that they are almost certainly not doing is to use cloud services to bridge the retirement of their mainframes and move everything back on-prem with Linux boxes in 5 years.
People often say that, but that literally has happened once, with GCP, and has no relation to how AWS and Azure do things. Please go and find an example in the 10+ years that AWS have been a big serious contender for the world's IT workloads of them jacking up prices.
Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper than yours.
Getting computers from the cloud companies is very different: most cloud products (aside from server rentals) are not in competitive markets.
Fast delivery couriers do not ship by sea.
It looks to me like the cloud providers will soon have FedEx nicely tied over a barrel, and those 'savings' will prove illusory .
Exactly, which is why it’s quite a bit cheaper to not have a building and staff right? The premium you pay for the cloud provider having them must be less than having them yourself, or nobody would ever move to a cloud provider.
In order for the cloud business to make sense, turn a profit and sponsor the kind of development into new products done by these companies, we can assume that the cloud services sold by these providers must have a very healthy margin compared to the cost of operating a datacenter full of resources.
The primary thing a cloud business can do to drive cost back down past what you could do with your own datacenter is to have better utilization of their resources by dealing with an average across a much wider range of workloads, or by selling spot instances that get killed if capacity is needed, but based on how things are priced it appears that this mainly just pads their margins.
For the customer, it only really ends up cheaper than on-premise is if: A) you need very few resources and do not already have anywhere to put a server or lack a good uplink; B) your use-case is extremely well-optimized for cloud (e.g. extremely bursty serverless where you average load is a tiny fraction of a single machine); or C) you are Netflix and can make a deal with AWS.
* edited from million (meant to type billion)
With modern open source tooling, ridiculously powerful graphics cards, easily accessible FPGAs and free data center hosting software like OpenStack, Kubernetes, and glusterfs you can replicate many of the features that made mainframes enticing many years ago.
It'll require a heck of a lot of work to get the same performance out of a custom built solution, but long term it's probably cheaper than relying on IBM.
I doubt that they'll be saving any money by moving to the cloud unless they're horribly inefficient with their contracts right now. Just moving away from mainframes alone should save a significant buck, but if they were planning on restructuring their digital infrastructure anyway then now is probably the best time.
Granted, the footprint per sim bay isn't huge (1-2 racks, usually, sometimes 3), but it can add up at the larger sim facilities. Several of the major airlines are running 30+ bays simultaneously, with United poised to come in with the most at 40+ in the near future.
In general aside from the boxes handling the graphics projection, most of the 'compute' is idle a lot of the time, even when the simulator is running. There's usually a big burst of work to prep and load the simulation into the simulator, then minimal (but very latency sensitive) chatter back and forth between the various simulator components and the backend compute as the simulation runs.
It... was an experience.
I imagine that is going to take quite a bit of work to migrate over to someone else's servers, as I doubt they're going to be like "sure, we can accommodate your decade and a half old, mission-critical, systems"
- Build middleware (REST API endpoint) in front of the mainframe that was 1:1 mainframe functions. It initially does nothing except auth and routing.
- Re-write/migrate all other software from using the mainframe directly to using the new middleware.
- Work to further restrict anything directly connecting to the mainframe aside from the middleware, including staff's terminal access by building management solutions (e.g. new UIs, new management middleware).
- Once the mainframe is completely isolated, start building drop-in replacements for business segments and then switch the middleware(s) to point at the new solution from the mainframe. This should be invisible to all external consumers. You can "hot test" it by having the API execute on the mainframe + new solution, and checking for 1:1 responses.
The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory). So you have to create very compelling alternatives using the new API middleware/web/apps, "export to Microsoft Excel," graphing, mobile support, and similar that a terminal cannot directly offer is clutch here.
There was a lot more complexity that that, but that's the gist of it. By the time I left, my team had carved out the bulk of the mainframe's services and had stood up over 50 microservices.
The hardest part about the above isn't the tech, it is the internal norms that you're fighting against
This is so true it's almost painful.
I have sympathy for those people. Not a fan of AS/400, but working in those terminals is incredibly fast once you know all shortcuts. Modern GUIs, especially using all those fancy frameworks, are much slower. If the alternative to the terminal is a new web application, it's extremely likely that it'll be harder to work with.
But one of the easiest way to retire anything, is to stop doing new work on the older systems. Any new project should go on the new tech.
Soon enough you will see the older systems gone. It won't happen in weeks or months, but 1 - 2 years is enough.
At one point the whole banking industry used to de-facto run on Java. Eventually people just started doing new work on Python. These days Java is just another tech there.
Just because you don't invest in a system doesn't mean it becomes obsolete.
Can their existing mainframe software and databases simply migrate into a cloud offering? If not, then if their planned transition has delays or technical hurdles, how much of that cost savings is affected?
If their cloud providers change pricing structures in future contract negotiations or if the amount of data they transfer in/out of each cloud provider location changes dramatically in order to better serve their customers, how much of that cost savings is affected?
It's clear that you can run a business with minimal amount of physical plant for servers. It's still not entirely clear to me that large established businesses can actually save money this way.
Let's assume for a moment that modernization is good. It probably is wrt mainframes. For every team that refused to modernize, you can now drive them to the 2024 date, absolutely no exceptions.
If it drops all the way to the bottom line, that would boost their 2022 profits by 35% and 2021 profits by about 45%. Seems like a decision you have to consider, even with possible failure modes.
Wikipedia says their net profit for 2021 was $75 billion, so your math seems off.
That said, it seems like $75B is wildly too high as well. Page 43 of their most recent annual report shows a consolidated net income for the year ending May 31, 2021 as $5.2B, making $400M still a 7.5% boost (if it all passes through).
I am not even sure where the $75.231B erroneous net income figure comes from, it is nowhere in the 2021 report:
* In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them.
* An extensive suite of complementary products are available from cloud providers, it's not just about being able to start a VM, in many cases you don't even need to because you've got Lambda, and the various managed services.
* Less training/hiring overhead, since people who can use AWS are pretty much a commodity at this point, whereas people who can manage a mainframe are an increasingly rare breed.
* Redundancy becomes easier. As does data locality, let's say Singapore declare all services delivered in Singapore must be hosted there. That's a lot easier to handle if you don't have to go and build/acquire a data centre first.
* The aforementioned 400 million dollars per year.
How many people run physical servers 'raw' anymore? I would hazard their current stack involves VMware, Hyper-V, Open Stack, or a combination of the three.
You can have an API system spin-up just as effectively in a private cloud as a public one.
Not saying it can't be done well but there's less incentive for a company whose main business isn't providing infrastructure to ensure they've provisioned enough resources to meet demand.
With cloud you can rely on more datacenters and, if your application is built right, it may end up being more resilient than what a mainframe setup would give you.
Downtime is a fuzzy thing. Whenever possible, I design my apps for graceful degradation - if the database becomes read-only, we can still operate in degraded mode. If we lose queues, some things will not work, but others will continue normally and many users will be completely oblivious to the alarm bells at the NOC (just kidding, there's no such thing). My SLA for full operation is much more relaxed than for degraded operation.
You could read HN articles you read before, for example!
Not to rely on this for the nines, but as a nice way to keep things useful when there is an outage or just slowness.
EDIT: I used to be like that, too.
However if they were to undercut UPS and others they might be able snap up UPS business, dropping the profit to $1, but on 200 million deliveries, and thus they'll make $200m instead, you the customer will have a cheaper delivery, and everybody wins
This is competition.
It fails when you get high barriers to entry and company colluding
I'm not being facetious.
Are they really going to close all their on-prem datacenters in the next 2 years? Maybe.
More likely, this is a way to build momentum around the transition to cloud. Not a bad strategy, and not uncommon from what I've seen in the enterprise.
With Amazon being a formidable logistics competitor, I can see why Fedex might be loath to fund a major competitor.
I guess I'll prepare for the inevitable apocalypse then...
It would be interesting to know how Fedex sees the risk/reward equation, and how Azure (for example) compares to a (IBM?) mainframe data centre in terms of reliability and uptime
‡ I'm not sure if there are any cloud neutrality solutions that permit this, but it is possible that FedEx have rolled their own.
Surely FedEx would have a metric shit ton of data?
Really, not the first example I’d reach for.
But when I spoke to a CTO of a F500 company. This was the take he had. He feels one of the major driving force is to reduce the carbon foot print of the company. His company is designing the system to have the crucial data on their own infrastructure but keep they other data & do the processing of cloud service. Instead of relying on 1 provider, they are using multiple. Also supporting some small local cloud providers.
This seems very relevant for a logistics company with a huge of fleet of air craft. Especially considering the upcoming carbon tax.
Also, from my understanding, it can take a tremendous number of distributed x86 systems to supplement the scale, speed and reliability of a mainframe. It's possible that unless FedEx gets away from managing storage arrays and hypervisor farms, they wouldn't have enough staff to operate the datacenters once they're full of enough servers to replace the mainframes.
Just like many other industries, the market for mainframe operators that want to earn less than technologists in modern stacks has been drying up. A lot of mainframe operators refuse to bring their pay scales in line with what developers in modern stacks are earning. Why take a pay cut to go work on a 40 year-old system when you can work on modern stuff with more money?
They keep the pay low and complain about a labor shortage to deflect from the fact that they're simply choosing to ignore market forces.
What sort of work is typically done on such machines these days.
They've been doing the same type of work for decades. Batch processing of records and transactions (payroll, insurance, banking, airline booking, etc.) IBM releases new mainframe models all the time, and typically companies will lease them and upgrade every few years.
When I worked at an insurance company, the mainframes were the core of their financials. They had around 8 of them in an active/standby DR site. The machines could do instruction-level error checking, meaning even a weird processor failure/error wouldn't impact the end transaction. Basically if you want to guarantee accurate handling of your money, you use a mainframe.
One common thing to see nowadays is the mainframe becomes abstracted by some other layer of API, such as a rest API. This has reduced the need somewhat for mainframe operators, but in the end you're still interacting with some COBOL that was written 30 years ago and somebody needs to understand it and the hardware. And the hardware is so different from traditional servers, you can't just pick it up and wing it. You really need someone who knows what they're doing.
A company then have two options:
(a) replace/pressure top/middle management to get costs down and processes streamlined or, (b) move to the cloud. Its easier (and possibly good for your resume) to move to the cloud and then a few yours later -- decide again.
So moving to the cloud is sometimes an indicator of an inflexible/stale/mature company.
But perhaps they're talking about engineering velocity, reliability, etc. Of course everyone knows Dropbox's story with this: https://www.datacenterknowledge.com/manage/dropbox-s-reverse...
Does that truly sound achievable, when they have the most difficult to migrate part of their mainframe applications still on their mainframe? The easy stuff they have already moved off.
In light of this, it's easier to outsource the network and data infrastructure to a provider that actually does it day in and day out and put the limited engineering resources to something that is more impactful.
Just reaffirms that quite a few companies do not want to run their own servers unless they absolutely have to.
Surely this is just one ingredient to a successful company and there are myriad factors involved. The core business of Boeing isn't to make planes its to sell them.
I imagine Twitter could also be built over a weekend, right? :-)
But yeah... old company, they might still have stuff written in cobol virtualized somewhere.
That seems implausible: where did they keep their mainframes before 2008?
Is that power. Or something else?
It sounds like Switch Inc. advertise their datacentres by Wattage. I am not sure how common that is.
It also advertises the square footage but you don't buy by the sq ft either
Usually in larger DC deployments, yes, you are quoted, metered, and billed by total power. The amount of space is (usually) an afterthought. There is more space than you can fill and remain under the power commit, which also correlates to total cooling load too.
They. Own. The. Data center(s).
Well, in some sense or another. They might not be completely paid for but you get the idea, there's got to be a certain amount of worthwhile assets that could be leveraged.
Too bad there's not any business executives who have any experience at making money from a data center itself.
I've seen lots of companies announce their intentions to move off of their mainframes, but I'm not sure I've ever actually seen one pull it off.
Related to that, one funny thing I remember from Robert Heinlein's "The Moon Is A Harsh Mistress" is that the Lunie protagonist didn't understand why Terrans bothered with insurance companies. Are there no bookies?
In the very short term. Data centers are expensive as hell to operate, but so is the cloud. I think colo would have been the way to go.
But labor costs keep going up and cloud computing costs keep going down.
So I don't see how this is short-termism.
I also have to keep reminding my coworkers that AWS gives us faster cores every few years but so far they always have the exact same amount of memory. So caching things (especially in process, but also on loop back) to save computation isn’t really a winning long term strategy. It’s never going to get better than what it does for you in the beginning. It will only decline in value.
They tried and failed?
Reverend Mother Mohiam:
They tried and died.
Given of course that everyone would have watched Father Ted. But the world would be a better place if everyone did.
Either there are a lot of Fortune-500 CTO's on Hacker News that know better than FedEx, or there is a deep-seated fear of the cloud, specifically PaaS.