A (0:02)

Starting in the early 2000s, reports emerged of abnormal numbers of PC motherboards with leaking or even popping capacitors. A victim said in a newspaper interview that he turned on his computer one morning and suddenly heard a loud pop like a distant firework. Other reports mention a weird fishy smell in the air. The OS goes black and the computer fails to reboot. Most inexperienced people have no idea what to do, except to bring it to a repair shop, open it up, and inside you might find that a little cylinder on the board had swollen or even burst. This happened a lot. Why? In this video, we look at the infamous capacitor plague. We will never know what exactly happened, but let's try. So let's start with the basics. A capacitor. What is it? A capacitor stores electricity, kind of like a battery. But unlike a battery, the capacitor discharges its electrical energy in a very short period of time, sometimes in only a few microseconds. The basic structure of a capacitor has two electrically conductive plates called electrodes, the anode and cathode, separated by a material called the dielectric. Eagle eyed viewers of the channel. I hope there are a few of you out there might recognize the term capacitor from videos on dram. And indeed it is conceptually similar, subject to the same laws of physics. Electrolytic capacitors are often called E caps, and I'm going to call it that too, because I hate saying the word electrolytic. The name refers to the electrolyte which serves as the capacitor's cathode or negative electrode. This electrolyte can be a solid gel or liquid. Simply put, an E cap is made up of two strips of aluminium foil with another very thin separator layer of porous paper or tissue in between. The paper and the gap between the foil strips are saturated with the electrolyte. The positive electrode or anode foil is coated with another layer of aluminium oxide. This oxide is the dielectric. The foils are attached to tabs which connect to the circuit board and the whole thing is rolled up in a sealed protective housing, usually also made from aluminium. The seal is to keep the electrolyte from drying out. There may be a vent at the top, so to release built up gas and preempt a messy explosion. While there are many types of capacitors and even electrolytic capacitors, like those made with tantalum aluminum, E caps are commonly used because they offer pretty good capacitance, as in they hold a lot of charge for their size and cost. A capacitor's ability to hold charge is correlated to the surface area of its electrodes and inversely correlated to the distance between said electrodes. So with an aluminium E cap, the thin separator paper lets the dielectric get super thin so so that the electrodes can be as close together as possible. Manufacturers will also make them work even better by etching the aluminium foil to increase the electrodes surface area. Nifty. A PC motherboard might have about 60 ecaps on it. Like why? What are they used for? A big use case is filtering. A power supply module receives AC power from the plug and turns it into steady DC power. AC power going into the power supply hits thing called a rectifier which converts it into DC power. The DC power that comes out is no longer alternating, but it is also not steady. We need to make it steady. So we use a capacitor to help smooth out the ups and downs. When voltage peaks, the capacitors charge up. When voltage dips, the capacitor discharges to compensate. Capacitors are also used to help smooth out power supply fluctuations going to chips like the microprocessor, A process called decoupling. The capacitor will charge up to absorb voltage spikes and discharge to boost voltage dips. I mentioned before that these decoupling capacitors are like drainage ponds for during floods or dry periods. And by the way, it's not just PC motherboards that have these eCaps but also boards for camcorders such as VCRs, televisions, stereos and LCD screens. The aluminium ECAP industry is a multi billion dollar industry and in 2002 is worth about 3 to 3.5 billion. Today various market analysts say it's worth about 5 to 6 billion. ECAP suppliers are mostly Japanese, Taiwanese or Chinese. And they produce literally billions of units each year. In 2002, 22.5 billion ecaps were made in Taiwan, 30% of the market. Perhaps because they are electrochemical, Electrolytic capacitors age and corrode and have shorter natural lifespans in general than other types of capacitors. About 10 to 20 years. The best way to see how far gone the capacitor is is to look at a metric known as equivalent series resistance. As the capacitor ages, that metric starts to rise. The capacitance will also slowly decline before dropping off at the end. This is a parametric failure because it involves parameters. Several factors can accelerate this aging process. The most significant being humidity and heat. Heat can cause the electrolyte in the ECAPs to dry up, so it is generally recommended that we keep them cool. Sometimes though, capacitors can fail in an extraordinary way. A catastrophic failure. A bulge capacitor has no capacitance for causing a short circuit on the board. An exploded capacitor might even cause the thing to catch on fire, and that is what started happening in the early 2000s. The reports of bulged or even exploded capacitors largely started in late 2002, but it seems like they began a bit earlier than that mid year. An electronics repairman in Utah named Gary Headley was quoted saying that he had replaced some 40,000 leaky capacitors on on almost 1,500 motherboards in a year starting in the summer of 2002. He said over 10 a day sometimes and they just keep on coming. Headley also claimed to see a significant increase in faulty Ecaps in Mitsubishi televisions, JVC VCRs and Sony camcorders. One of the journalists covering these failures early on was Kerry Holzman, who in October 2002 started posting online about leaking capacitors in various Taiwan made motherboards like 8bit Asus and so on. Holzman has a popular YouTube channel and in one video he covers his experience in finding and bringing this stuff to light. I think it's worth a watch. The story behind these failures was then reported by Dennis Zoghbi in the September October 2002 issue of the Passive Component Industry magazine. He wrote that a materials scientist working for the Japanese electronics company Rubicon Corporation left to go work for the Taiwanese company Luminous Town Electric's factory in China. The scientist replicates the formula for Rubycon's P50 water based electrolyte. Luminous Town then uses it for their aluminum ECAPs and it seems to work fine for them. But then some of the scientist staffers defected from Luminous Town with the electrolyte formula. They then either sell it to Taiwanese ECAP companies directly or or to an electrolyte provider who then resells it. The problem was that the defecting staffers did not have the whole formula. There were missing additives that cause the ecaps to suffer hydrogen gas buildup until they rupture or break. In the following issue of the magazine, a rather amusing clarification reiterates that Luminous Town did not make any of the bad electrolyte circulating through the industry. Luminous Town did not deny the story of stealing electrolyte formulation from Rubicon though, which is weird. Zoghbi's story quotes anywhere from five to 11 Taiwanese producers being affected. In a 2003 interview with the Toronto Star, he said, I think anybody who uses contract manufacturers that outsource to Taiwan was affected by the problem. Zoghb doesn't note in the original magazine article when this whole thing occurred, but in a 2003 interview with the Santa Fe New Mexican newspaper he says that it happened sometime in the middle of 2002. I've seen mentions of 2001 and even 1999 in other publications, but 2002 looks to me most accurate. The IEEE Spectrum magazine did a story on it at the time and noted that the broken ecaps bore generally unknown brands like Daie, Choyo and CIXI or are unmarked. A few showed the name of Jackcon Capacitor Electronics. Jackcon's managing director replied that they had not made capacitors for motherboards in the past two years, which implies that someone used the wrong ECAP for the job. In a later newspaper interview, Zoghb says that he was tipped off on this story from a Japanese contact and confirmed it with other anonymous industry sources. IEEE Spectrum wrote that a well placed Taiwanese source largely confirmed the story too. I reached out to Zogbee via email for a chat but didn't get a response anyway. Zoghb's report noted at the end several computer makers commenting that they had indeed seen some issues with Taiwanese ECAP failures. IBM, which back then still made PCs, was willing to go on the record. IBM spokesperson Ray Gorman said that a small number of desktop PCs were returned for repair after short circuits induced by a capacitor failure. He added the problem was miniscule, affecting less than 1% of computers. Maybe true, but less than 1% can still add up to a lot. Gartner said that the world PC industry shipped 148.1 million PCs in 2002. Less than 1% of that can still cause major disruption to workshops and the like. A second manufacturer to admit to an ECAP problem was the Taiwanese motherboard maker apit. They told the press that they were switching from Taiwanese suppliers to Japanese suppliers as a response, as a thank you for their honesty. 8bit got sued for selling defective capacitors as early as 1999 and so far as I can tell they that is why most people peg the start of the capacitor plague to 1999, though I have not seen real reports of such being that early. Anyway, abit settled and repaired the bad boards. The cost of doing so may have contributed to their later exit from the motherboard business in 2006 and practical liquidation thereafter. Interestingly, the IEEE Spectrum store even named the electrolyte company accused of buying the flawed and stolen formula Lian Yang in Taichung. Lian Yang denied it. They said that the accusations heavily damaged their business and pointed out that the suppliers of several bulge capacitors never bought their electrolyte. They called the Rubicon story fud spread by the Japanese to win back business from Taiwan. One of Taiwan's biggest capacitor suppliers, Luxon, issued a statement anyway, blaming Lian Yang Yang for all the drama, emphasizing that they never bought any electrolyte from Lian Yang. They added, Luxon definitely understands that the electrolyte is one of the most important materials for aluminium electrolytic capacitors. In order to ensure our reliability and innovative technology, Loxan always develops the electrolyte by ourselves. I did a few Google searches but couldn't find out what Lian Yang is up to today. But without the Chinese name, tracking them down can be hard. Lian can mean anything. I bet they changed their name or went out of business. Probably the latter. Luxon's remarks reinforce the point that the electrolyte is not just a simple mix that you pump into the capacitor at the last second. Rather, the electrolyte is a critical trade secret that sits at the core of the whole product. It determines the capacitor's operating conditions and temperature, voltage safety and more. These are complex formulations with many chemical components. The Rubicon story indicated that something had been missing from the formulation, and in 2004, the center for Advanced Life Cycle Engineering at the University of Maryland was asked by a capacitor maker to do an analysis for publication. Their report, which I found rather challenging to read, indicated that the electrolytes of the bulging capacitors were missing a critical additive, generally called a depolarizer. Depolarizers can help reduce the generation of gas within the E cap. In this case, the lack of a depolarizer caused the electrolyte in the bad ECAPS to be more alkaline or basic than it should be. Due to this and the fact that the aluminium oxide dielectric lacked other known protections, like phosphate ions, the dielectric dissolved into the alkaline electrolyte. This caused the middle dielectric layer to get thinner, which in turn causes the capacitance to get unnaturally high. A key leading indicator of bulging. This exposed the underlying aluminium foil to reactions where impurities in the foil can react to create spare electrons and those electrons then in turn react react with the hydrogen ions in this water based electrolyte to create hydrogen gas. Talk of the capacitor plague tends to have the Rubicon industrial espionage story attached to it. We will never really know the truth, of course. There's nothing about it that makes me think it can't be true. In fact, I'm pretty sure something like it actually happened. Industrial espionage is a real thing. However, I am skeptical that the whole plague can be blamed on a single defector from Rubicon. I'm more apt to believe that this was an extended series of unconnected challenges faced by the capacitor industry as a whole. Why? First, these were quite widespread, and people are too easily attributing every capacitor failure to the plague and thus the Rubicon story. In an earlier hacker news discussion about the capacitor plague, there was a mass of MeToo comments testifying that the plague hit their audio equipment, Samsung monitors, and famously Xboxes. That makes me skeptical. The other stuff we can't trace, but we can with the original Xbox, which was released in November 2001. That console is known for a bad capacitor that blows about six, seven years in, and there are articles teaching users how to preemptively remove it on their own. People are apt to blame this bad capacitor used for a clock on the plague, but this capacitor is a Powerstore Aerogel made by Cooper Industries, an American company based in Texas. Not only is this cap American, it's not even an aluminum ECAP. Per a July 2000 story in Everyday Practical Electronics magazine, the Powerstore Aerogel is a supercapacitor with its electrodes made from carbon Aerogel foam, not aluminum foil. I doubt that it uses an aluminium oxide dielectric, but feel free to comment if I am wrong. Considering how new it was, I'm not surprised that these puppies didn't last very long. So we have an unrelated case of bad capacitorship, but since the timing seems to match up, people blame it on the poor dude at rubycon. And anyway, I think this happened a lot more than we think. And speaking of timing. Second, the timeline is all weird. The Rubycon thing happened in mid-2002, per Zog B's recollection. Again, there are some unsourced statements that it might have happened in 2001 as well. I buy that less Even if we're being charitable and assume it took months to discover the bad batch, the common sentiment at the time was that a bad ECAP would reveal itself within a year. Perhaps even less. Intel in the original 2002 report said a poor electrolyte ECAP would fail in as little as 250 hours of operation. So how can such sleeper ecaps keep blowing up into 2007? In late 2005, another round of bulging capacitors hit the news. Apple and Hewlett Packard computers were affected, but Dell most of all. From May 2003 to July 2005, Dell shipped some 11.8 million OptiPlex PCs to mainstream and corporate customers with faulty capacitors that were 10 times more likely to bulge. Dell people apparently knew of this and tried to hide the issue from customers, and thus they got sued for it in 2007 and spent $300 million on replacements. All of the bad capacitors came from a single manufacturer, Nichicon in Japan. HP put them on blast. When the news first came out about this in September 2005, Dennis Zoghbi wrote that industry insiders told him that the Nichicon bad products were made by a Taiwanese subcontractor. OK, but three years earlier, during the 2002 bulging capacitor incidents, Nichicon claimed that they had no plants in Taiwan. Sure, it's not the same, but something about the timing and the circumstances doesn't sit right with me. So if it is not the Rubicon scientist, then why are all these capacitors failing? Going back to Dennis Zogby's 2005 report about the new round of capacitor bulging, he wrote, my two electrolytic scientists, colleagues in the industry felt that using an electrolyte that has been on the shelves since 2002 in Taiwan was unlikely and that there may be a problem with using water based electrolytes to decouple new high speed microprocessors. Now this makes sense to me. These new low ESR aluminium ECAPs with water based electrolytes were only introduced in the late 1990s. Such caps were brought in to handle increasingly power hungry computers. The early 2000s were also a time of intel and AMD aggressively pushing single core clock rates. But with the end of denard scaling, these chips were consuming a lot of power and getting very very hot. I note that the failed OptiPlex PCs from Dell were running Pentium 4 chips which were infamous for this. So my headcanon is increasingly hotter. Microprocessors at the turn of the century heated up these new capacitors to such an extent that their water based electrolytes simply boiled, creating gas that popped their containers. When AMD and Intel Transitioned to multicore CPUs starting in 2005, the internal temperatures scaled down. At the same time, capacitor makers got the hang of making these newfangled aqueous electrolytes and together this finally ended the capacitor plague. One of the problems Zogby pointed out in 2003 was the increasing complexity of the supply chain. He talked about brand name companies outsourcing parts like ECAPs to no name suppliers, offering the cheapest prices. A valid concern. But there's another thing to counterfeits. Even in the best of times it could take several weeks to get a set of ECAPs from manufacturers abroad. But when supply gets disrupted for whatever reason, that puts pressure on the oem, because what customer is okay with waiting so long for a tiny thing that costs less than $5 each? So the OEMs turn away from an authorized distributor and buy something off the market with only a quick check of the brand. The issue is that the capacitor is a counterfeit and it won't last anywhere near as long as the real thing. One intriguing 2014 study of a set of counterfeit Nichicon ecaps discovered after the March 2011 earthquake interrupted the legitimate supply chain found that the counterfeit water based electrolyte lacked enough ethylene glycol. This lowered the electrolyte's boiling point, making it unstable at higher temperatures. Makes me feel like this sort of error is not all that uncommon. All in all, this is a weird story and was a challenging one to tell. I've seen a number of online commenters insist that all the company's official statements are lies, that nothing can be trusted, and that the stolen formula story is the truth. If that's your insistence, there's nothing I can say to convince you otherwise, but I mostly reckon that the Rubicon incident story is just a fun urban legend riding along with the real technical issue being worked out over the years by the capacitor industry. Aluminium ecaps with water based electrolytes are still around and there are still longevity concerns, but they still offer the best capacitance for their size and and price. And it seems like the vendors have largely worked out the kinks. The little round can on the circuit board can go back to being unnoticed. Alright everyone, that's it for tonight. Thanks for watching. Subscribe to the Channel, sign up for the Patreon and I'll see you guys next time.