And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything?
How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done?
And now you see why banks have such terrible IT.
(One of my mentors actually works at BofA, and says he only does it because he gets to work 6 hours a day, gets a VP title and a ton of money, but nothing ever gets done)
It's not just infrastructure. Quality, innovation, speed, efficiency, etc. To whatever degree you believe the old 10x programmer meme, believe this: Change that constant 10 to any value for many employees at once at a company and the impact is staggering.
Good people don't think to they are too good for these types of companies. They think the companies are too constraining and place a limit on their ability to realize dreams or their potential to perform.
They don't prefer to work with less passionate peers. Not because such are lesser human beings to be avoided. Rather, it gives them another edge. Constant, motivating, cross-reinforcement during discussion, doesn't just inspire it creates action.
Finally, most of these companies don't have tech as their core business. That correlates very directly to the fact that the most senior and influential people in the company will care less about you, being willing to partner on ideas, etc. It's not personal, Banks think bank stuff is most important and anything else is less important, even if it's a critical operational component of the business.
You're the best engineer willing to work in a black box, a niche, in isolation from your peers with constant pushback.
If the answer to that challenge was one of "too hard", "it works for now" or "too expensive" that's still a strategic failure that impacts their core business now.
This is why sometimes absurd amounts of money are paid to keep the old system going as it is. It's "safer" than to move to a new one with the associated risks and teething issues.
This is not really related to costs. It's related to culture. Costs and quality of software, including infrastructure, range by orders of magnitude per theoretical quanta of software.
We don't really know how to fix this.
The only people who think auditors are bad are the ones trying to pull something an auditor would catch.
https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...
If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.
I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.
Tangentially related, I highly recommend the movie "Out of the Clear Blue Sky." Cantor Fitzgerald was a bond trading firm at the top of one of the twin towers and lost every employee who was in the office on 9/11. Incredibly, despite losing the majority of their employees and despite losing almost all of their trading infrastructure, they managed to resume operations in time for the bond market's reopening 48 hours later.
https://finance.yahoo.com/news/every-wells-fargo-consumer-sc...
You can't really roll back say 10 minutes of transactions, so are you maintaining 2 parallel systems? How do you keep them perfectly in sync?
This isn't my area of expertise by a long shot, but it occurs to me this is probably hard, especially when your codebase started in the 60s, and has been accreting ever since.
And if anyone ever figures out that isn't the way it is, and that the numbers are not representative of anything of substance? If nations refuse to honor the claimed 'transfers' done through these rickety electronic systems? It would make for an interesting few days.
Also, failover is hard. Few companies outside of a few larger Internetz companies can really do it well.
Im employed by WF and even Im a little bewildered by the fact there doesn't seem to be any redundancy implemented somewhere.
Big companies tend to defer risk. Managers and project leads want to start new projects rather than upgrade existing infrastructure. Combine these forces and sometimes you get a catastrophe.
Are you sure you're not one?
After all their logo is that of a stage coach aka the wild Wild West(robbers, thieves, etc). They do not hide who they really are.
Wonder if it was the switchgear that failed. Amazon uses custom firmware in its switchgears because this happens so often (Superbowl 2013 etc.)
https://perspectives.mvdirona.com/2013/02/the-power-failure-...
https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...
https://perspectives.mvdirona.com/2017/12/when-you-cant-affo...
great job guys
- for instance, in cali: https://www.golden1.com/
- high interest accounts: https://www.bankrate.com/banking/checking/best-checking-acco...
- good rate online bank: https://empower.me/
get better service, lower/no fees, and good convenience without all the economy-damaging selfishness and hubris.
when i choose a credit union, i optimized for two things:
- is there a branch close to work or home? this is mainly to develop a personal relationship in case i ever needed a loan. i'd deposit checks and withdraw cash in person occasionally.
- good rates on interest bearing accounts (at least 1%, but often >2% apr)
if you work for a larger institution, disney for example, joining their credit union is convenient.
https://www.nerdwallet.com/blog/banking/best-high-yield-onli...
So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ.
Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to recover? What if a data center was nuked? What if two data centers were nuked?
How much redundancy is in the system? Are there third-party agencies that track private bank ledgers? How hard is it to take them out too?
Ever since I read "The End of Alchemy" (a great book, btw) this thought has haunted me.
If you are talking about actual nuking then the story might be different. Not all backups can survive an EMP. I expect the biggest problem would be getting people to care about bringing the system back up. I think food and shelter would be of primary concern.
Although I'm not sure if it could withstand a nuke, its in a mountain, so, maybe?
What would happen if there were no records? Surely there’d be lots of people making significant deposits in between snapshots. If you’re “paperless” I’m not even sure how you’d reconstruct your balance.
Would certainly be time consuming, and in this case time would certainly be money.
I'm genuinely curious what the process would look like, if it even applies. Where would you even start if the bank just suddenly says "you have $0"?
Please be advised that the Wells Fargo Gateway is currently unavailable.
We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We're working to restore services as soon as possible.
We apologize for the inconvenience as we continue to work on a resolution.
We will update you no later than 3:00 pm, Eastern Time.
https://en.wikipedia.org/wiki/Hanlon%27s_razor
recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor.
from the reddit:
"throwawayfordays75 1399 points 8 hours ago*2
Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. "
At this point im wondering what "utility work" was happening.
"everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on."
Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now.
Or have we been so focused on never cold booting a rack of servers that we haven't spent any effort on foolproofing of cold booting a rack of servers?
[Edit: answering my own question] apparently these exist and are called Managed PDUs. Can we deduce WF doesn't have them?
This does not impact Stripe customers at this time. Specifically, the Stripe API and Instant Payouts are functionally normally.
(We're aware of the issues Wells Fargo is experiencing, and have an incident response spun up internally.)
[0] https://api.twitterstat.us [1] https://help.netflix.com/en/is-netflix-down [2] https://developers.facebook.com/status/dashboard/
this is what the merry band of racist edgelords have fixated on?
Seems like it would be more /g/'s thing.
Pardon my immense ignorance!
This thread has primarily focused on redundancy and software architecture. That could be the case, but there is no better way to fight proxy war via hacking of banks. It’s a domino effect... pull your money out of the bank, the bank calls loans, then there is credit crunch, investors lose confidence in the stock, value lost etc... the enemy has secretly inflicted civilian problems across the economy. Distrupting the banks and the flow of money can lead to a revolution... or take your mind off of America’s enemy or diverting energy else where.
Oh wait...
