TSMC: Outbreak of Malware That Triggered Delays & Losses Caused by Software for New Tool
by Anton Shilov on August 9, 2018 3:00 PM EST- Posted in
- Semiconductors
- 16nm
- 10nm
- TSMC
- 7nm
TSMC announced this week that it suffered a computer malware outbreak, resulting in a roughly 3 day outage for parts of the fab while systems were restored. As a consequence of the downtime, the fab expects certain shipments delays and additional charges. Specifically, because of the interruptions and costs, the company’s Q3 revenue and gross margin will be 2% and 1% lower than anticipated respectively. TSMC later clarified that the outbreak was caused by “misoperation” during the software installation for a new piece of equipment.
What Happened?
TSMC’s personnel set up a new manufacturing tool on Friday, August 3, and then installed software for the device. The machine was not isolated and confirmed to be malware-free before connecting it to TSMC’s internal network. Consequently, the introduction of a malware-infected machine to TSMC's internal production network allowed the malware to quickly spread and infect computers, production equipment, and automated materials handling systems across TSMC’s fabs.
According to the chipmaker, the malware was a variant of the WannaCry ransomware cryptoworm. WannaCry, though over a year old at this point, still has the ability to propogate among any remaining unpatched systems, which is what happened here: the malware infected Windows 7-based machines “without patched software for their tool automation interface.” As a consequence, the affected equipment either crashed, or rebooted continuously, essentially being inoperable.
TSMC has been stressing that not all of its tools and automated materials handling systems were affected, and that degree of infection varied by fab. The company had to shut down infected equipment and apply patches. By 2 PM Taiwan time on Monday, 80% of the impacted tools had been recovered and TSMC said that it would mend all of them by Tuesday.
The Impact
Since the said tools are located across multiple fabs and are therefore are used to process wafers using a variety of process technologies for different customers, it is evident that the outbreak affected delivery schedules for many chips. As a consequence, the company had to notify its customers and reschedule their wafer delivery dates. Some of the delayed wafers will be delivered not on Q3, but in Q4, thus affecting product launch plans.
None of TSMC's well-known customers are currently commenting on the matter, but this event has occured with what's widely believed to be the ramp-up periods for new chips from Apple and NVIDIA. Since at least some of TSMC’s production tools were offline for four to five days, it is evident there will be impact, though it is hard to estimate how significant it will be.
What remains to be seen is how several-day outage of numerous semiconductor production tools is set to affect TSMC’s customers in general. After all, 2% of TSMC’s Q3 revenue is between $169 and $171 million and that is a lot of money. We will likely learn more about the effect of the malware outbreak in the coming months.
(ed: As an aside, I find it very interesting that this entire episode was essentially happenstance, rather than some kind of targeted attack as would typically be the case. WannaCry is over a year old and is self-propagating; so as a proper worm, it goes wherever it can, whenever it can. In fact with the release of patches over a year ago, WannaCry's primary function is done. So for TSMC this is the IT equivalent of stepping on a landmine from a long-forgotten war, and reinforcing the fact that advanced malware can be dangerous to the public long after it has done its job. -Ryan)
Related Reading:
Facebook
Twitter
RSS
41 Comments
View All Comments
DrizztVD - Thursday, August 9, 2018 - link
As an ex-security researcher, I want to pull my hair out reading this. My only response is: you deserve it TSMC.I cannot understand how companies can be so lax with their security policies. It shows how little management knows about security and how they will prioritise short-term profits and just take a head-in-the-sand approach and not try to update their systems for the slight cost of maintenance downtime.
It tells me that there are likely huge security holes in TSMC infrastructure that a little bit of social engineering will be able to exploit to copy the latest processor designs directly off of their intranet. The way around this is that customers should work in clauses that automatically brings in fines if the security of a manufacturer is not up to scratch. We even need to see governments bring in proper fines for information leakage hacks to incentivise companies to stop being so lazy about it.
close - Thursday, August 9, 2018 - link
Sorry to break this to you but as someone who worked in security and with security people extensively I can only say that these are the people with the most narrow field of view I have ever met. They also have a single point of view and imagine it's the only one. They are always the ones "willing" to completely bog down a business with "security" without understanding that there's a compromise.I always expect that one day one of these people will just say: "Quick, disconnect every device from the network, cut the power, pour concrete all over them, lock the doors and go home; there, perfectly secure from hacks, (my) job well done". Or "you got a malware, I'm sure anyone can just use some social engineering thingy and steal all your designs, your monies, your children".
The business has to make a compromise sometimes and take a risk. Sometimes that bet doesn't pay off. Most of the times it actually does. And sometimes the fix could be just as risky or close to. The deeper you are in the field, the less likely you are to have any kind of perspective and vision.
Yes, it was a fkuc-up, yes it turned out bad and expensive, yes it could have probably been handled better, and maybe even completely avoided if the cards fell just right and the conditions were perfect. But no, try as you might you will never see it as it is. You will always see "oh, they didn't apply every single patch, didn't put every computer on a separate network with penta-factor authentication, oh [something else here that will definitely put a stick in the business' bicycle wheel]".
For one you fail to understand this is a private company and it's the customer's problem to complain, not your's or the governments. Not in this case anyway. As an "ex-security researcher" you should know that these vectors are as different as they get and they are treated in completely different ways. Social engineering is treated differently because there's really no perceptible downside to provide this education as opposed to possibly crippling production systems with a patch.
Roland00Address - Thursday, August 9, 2018 - link
I am sure you are both familiar with Richard I. Cook, MD's famous 18 rules on "How Complex Systems Fail." If you are not just google it for it is a very famous relatively quick paper on this subject.Samus - Thursday, August 9, 2018 - link
LMFAO I was thinking the same thing and reference Cook's rules all the time in IT. At the end of the day, security flaws almost always come down to money, or the lack of spending it. That isn't the IT departments fault. In many cases, management has been warned months or even years in advance that shit is eventually going to hit the fan if they don't do something about it.mapesdhs - Friday, August 10, 2018 - link
And the last thing we need is govts sticking their noses in, given their dreadful track record in managing security. Most govts can't even secure their national borders.FunBunny2 - Friday, August 10, 2018 - link
yeah, get rid of the FDA, which secures our drugs, while your at it. corporations are stellar at protecting patients.DrizztVD - Saturday, August 11, 2018 - link
Geez, you seem to think you're very smart. But, if you really were, you'd know that computer security firms are constantly in communication with their clients to figure out the details of a compromise between security and economics. That's really security 101, no great insight from you there.As for patching the Wannacry vulnerability, that is a security patch which TSMC rolled out the moment their systems got infected. Security patches are designed to be implemented with the lowest risk to productivity, and in this case it appeared to work right out of the gate for them. There is no excuse for not installing existing security patches, and any system without security patches is prima facie hackable with commonly available tools - free tools like Kali Linux.
CajunArson - Thursday, August 9, 2018 - link
Online Forum Comments: OMG! Spectre Sub-variant 4.A[2] exists! APOCALYPSE!Real World: We ran our whole internal network with totally unpatched Windows 7 systems and got hit by a more than one-year old worm that came preloaded on something we bought and didn't check.
Alexvrb - Thursday, August 9, 2018 - link
The part where you wrote "Real World" should be replaced with "Offline systems with no access to the internet that are generally secure from outside attacks unless we F up".For the ACTUAL "Real World": If you're on the interwebs, yes, you SHOULD be concerned about newer threats. Derp.
edzieba - Friday, August 10, 2018 - link
"Offline systems with no access to the internet that are generally secure from outside attacks unless we F up".No offline system is 'generally secure'. This is not some surprise revelation, just being separate from the internet does not magically bestow security.