Research Topic: Which Battery Will Do?
When Battery Testing Goes Bad – Consumer Magazine
First posted 12 October 2011, updated 27 November 2011, 19 January 2013, 1 September 2013, 7 January 2016, 11 September 2016, 2 October 2017, 25 March 2021.
The point of a good test is to allow buyers to make an informed decision about what is best to buy. Consumer has failed to do this. In its October 2011 issue Consumer published a rechargeable battery test titled "On and on ..." by Paul Smith which drew flawed conclusions from the report's own test data about which NiMH batteries are the best. This resulted in poor quality batteries being recommended by Consumer.
I've taken a closer look at Consumer's results for two of the batteries included in their test – the Energizer Recharge and the Sanyo Eneloop – and drawn my own conclusions about what Consumer's results actually mean. I'll also explain how they should actually have tested certain battery characteristics such as self discharge.
The first problem with the Consumer report is the Consistency score, which accounts for 30% of the total score for each battery. Their idea of "consistency" for a battery is "measured by the amount of running-time the battery loses over its life. A high score is good." (Italics in original.)
Actually no. A high score is bad.
Consumer's test procedure defined the end of life of their batteries to be when "the battery capacity is reduced to 50 percent of its starting charge", by which they probably mean 50% of its initial capacity. That's OK – the test has to finish sometime – but the article claims that the Energizer Recharge was "the most consistent performer, losing only 21 percent of its running-time by the end of its life."
How is it possible for a battery to lose 50% of its capacity but only 21% of its runtime?
Clearly the discharge did not involve a constant current discharge, which would always give 50% runtime for 50% capacity. This is an immediate concern with the test method because it's the simplest way of measuring capacity. If the discharge current is fixed, the capacity can be found simply by measuring the time taken for a battery to discharge down to a pre-selected termination voltage (normally 0.9 V). Any other method requires recording the battery voltage as the battery is discharged through a fixed resistance then calculating the capacity from the resulting data, or by measuring the current as the battery discharges and integrating over time to find the capacity.
The simplest way to explain the Energizer Recharge result is if the test involved a constant resistance load, for example, by using a 1 Ω resistor. The current the resistor draws will depend on the voltage of the battery according to Ohm's Law, V = I * R (or I = V / R). Because the resistor is 1 Ω, V = I. The battery condition deteriorates during the testing, so at the end of the test the battery cannot deliver the same voltage as when it was new. This is because the battery's internal resistance increases, meaning any load placed on the battery sees a lower voltage because some of the battery's voltage has already been dropped across the battery's increased internal resistance. Less voltage means less current, and because the current has decreased, runtime is extended compared with what it would have been with a constant current test. A standard resistor across the battery terminals allows the calculation of capacity using a voltage logger and Ohm's Law.
Ideally a battery's voltage would not deteriorate at all over its usable life, and like a constant current discharge it would also always give 50% runtime for 50% capacity for any given resistive load. It would always perform like a new battery except for not lasting as long. Unfortunately Consumer would give that ideal battery only 5/10 for Consistency.
In other words, Consumer's Consistency figure is actually a Crappiness figure. The higher the Consistency figure the worse a battery performed, because the extension in runtime is a direct result of an undesirable lower voltage.
From Consumer's results the Energizer Recharge got 2455 mAh at the start of the test with an initial runtime of 138 minutes, meaning an average 1.07 A discharge rate. 50% capacity at the end of the test, or 1227 mAh, with a 21% lower runtime, or 109 minutes, means an average of just 0.675 A. The significantly lower average discharge rate at the end of the test is a direct consequence of the lower voltage that the battery can maintain in its aged state.
This is not a battery in a good condition. Any normal capacity test would have found it to have abysmally low capacity because of its large voltage sag dropping its voltage under load to less than the normal 0.9 V termination voltage very quickly. It would have been rejected by many smart chargers long before the test was terminated because of its greatly increased internal resistance.
Compare that with the Sanyo Eneloop. The Eneloop got 2068 mAh at the start of the test with an initial runtime of 111 minutes, which works out to an average 1.12A discharge current. 50% or 1034 mAh at the end of the test works out to an average 0.93 A at the end of the test. This seems pretty good for a battery so well tested that it only has 50% of its original capacity left. Whether it was good enough to still use in the real world would probably depend on the exact purpose. Either way, those average discharge currents are better and considerably better, respectively, than the Energizer Recharge.
The last column in this table, Final Current % of Initial Current, gives a better idea of the state of the batteries than the figures Consumer presented, and are roughly the opposite of Consumer's Crappiness figures. Instead of doing these calculations, the Crappiness figures can be corrected directly using 50/Crappiness.
What difference does it make to Consumer's Overall Scores using these corrected figures contributing that 30% of the total score instead of Consumer's bogus Crappiness figures? The Energizer and Sanyo are no longer running neck and neck.
The Energizer Recharge got the lowest Corrected Consistency score (the highest Crappiness score) in the test but was summarised by Consumer as having "No obvious bad points." Clearly Consumer does not understand what a bad battery looks like.
Most of the low self discharge (LSD) batteries in Consumer's test are pretty consistent – as indicated by low Crappiness scores – so were incorrectly rated too low in their Overall Score. Clearly, the more consistent batteries are the ones that suffered less voltage loss and are thus able to maintain a higher average current over the battery's life.
This is a very serious failing of the Consumer test.
Runtime is certainly something to take into consideration with a rechargeable battery, but how justified is a simple comparison of runtime retention, especially when it's over the life of batteries that achieve considerably different numbers of cycles? For example, is the Energizer Recharge retention of about 80% of its runtime after 105 cycles really better than the Sanyo Eneloop retention of 60% of its runtime after 314 cycles? Directly comparing their end-of-life performance is crazy; the Eneloop got three times as many cycles!
What would have happened to the Eneloop's runtime after just 105 cycles? As it turns out, not a lot. Accelerated test data from Sanyo (something more battery makers should be unafraid to publish!) shows Eneloops are still going strong with roughly full capacity up to about 200 cycles, a point when all of the other batteries in the Consumer test were already dead or well on their way out.
The capacity of Eneloops also actually improves a little in their first few cycles. (My own testing – see "Update 20 June 2007" under the AA detailed notes – shows that after half a year of use they get up to 2116 mAh.) The capacity of ordinary (non-LSD) NiMH batteries starts decreasing right from their first cycle. The Energizer Recharge would have started deteriorating quickly after just 50 cycles, and had less capacity than the Eneloop after only about 60 or 70 cycles. How can Consumer consider that a more consistent battery?
This graph is not from the Consumer test data (which they did not publish with their report). It is based on Sanyo's fast cycle test data, and is supported by fast charge data from SilverFox and others on CandlePowerForums. Remember, Consumer thought these two batteries rated only 1% different from each other in their final scores, and because of the serious doubt that a 0.9 V termination voltage was used it's likely that the Energizer Recharge would actually have achieved significantly fewer cycles than Consumer says it did.
From the graph, since the capacity of the Sanyo Eneloop is basically unchanged at 105 cycles (the life of the Energizer) it's reasonable to assume that the Eneloop's runtime will also not be significantly decreased at that point. 80% of the Energizer Recharge's original runtime would come in at less than the Sanyo Eneloop runtime by the time the Eneloop had done 105 cycles, even without considering the very undesirable loss of voltage causing the Energizers to be crappy batteries by that stage. If Consumer was running a fair test then the Eneloop – and probably most of the good LSD batteries – would have rated at or near 100% for consistency at 105 cycles.
The basic point that can be seen from this is that if the Eneloop batteries had just been thrown out after only a third of their cycles they would have rated higher, while still completing the same number of cycles as the Energizers. That's absolutely crazy! Clearly Consumer does not understand what a good battery test looks like.
This is another serious failing of the Consumer test.
Let's see how this changes things at the point the Energizer Recharge got to.
The simple truth is that having a little bit extra runtime based on a battery's capacity when new is of no real value if the battery quickly become unreliable. If extra runtime when batteries are old is directly due to a sagging voltage under load it's very dodgy to claim that extra runtime is good.
Consumer paid lip service to cost by listing the prices of single batteries, but there's strangely no indication whether this contributed to the final score for each battery, and no attempt was made to compare the cost effectiveness of the batteries over their life. Not surprisingly Consumer did not consider the cost of frustration with crappy batteries either, or how bad batteries would just be left in a drawer (which doesn't give a good return on investment).
There's also a problem with the price Consumer listed for a 4 pack of AA Sanyo Eneloops. They are commonly available nationwide with free delivery for $24.99 from Dick Smith
Unless... Consumer has used Dick Smith
*Assuming no increase in cycles from 3rd generation Eneloops.
The Energizer Recharge cost 3.7 times as much per cycle as the Sanyo Eneloop (and remember the graph in the Runtime and Cycle Life section above showed the Eneloop had a higher average capacity).
These Cents per Cycle figures do not include the cost of electricity required to charge these batteries, which is almost negligible. At just 23.5 c/kWh (NZ$) it costs less than 0.1 cents to charge even the highest capacity battery, while still allowing for charging inefficiency of the battery and the power the charger itself uses. The Duracell Active Charge (a LSD battery) had the second lowest cost per cycle. The Vapex was one of the cheapest batteries in the test but the most expensive per cycle. Paying extra is no guarantee of quality.
The Sanyo Eneloop offers the best value for money; it's sad that Consumer couldn't bother to highlight how much better they are for value.
This is yet another failing of the Consumer test.
Self discharge is the tendency of NiMH batteries (and other sorts of batteries as well) to go flat while sitting around doing nothing, whether on a shelf, down the back of a sofa, or in a digital camera hidden away in a drawer. It's a problem because when they're needed, a battery can be completely flat because of self discharge (assuming the camera doesn't have a small parasitic discharge, which would flatten it more quickly). Particularly bad batteries can lose most of their charge overnight, meaning they always have to be charged immediately before use. It's like a car with a hole in the bottom of its fuel tank.
Low self discharge batteries – unfortunately abbreviating to LSD – were invented to address the problem, with the first being the Sanyo Eneloop. At last there was a NiMH battery that didn't need to be charged immediately prior to every use!
Their LSD ability worked so well the Eneloop revolutionised the rechargeable battery market. The concept has been so popular that all battery makers – with the notable exception of Energizer – now have LSD batteries amongst their products. LSD batteries are normally sold as "ready to use" or "pre-charged" because thanks to their LSD ability they can be sold with a partial charge. Normal NiMH batteries are completely flat when bought.
In the comments for the report on the Consumer web site, doubt has been raised about the validity of the self discharge test results. Many battery users are familiar with how quickly non-LSD batteries self discharge, and the 12 week duration of the self discharge test should have been long enough to clearly show the benefit of LSD batteries – test data shows non-LSD batteries typically have roughly twice the self discharge at 3 months that LSD batteries have. Consumer's results did not show that. Why not?
The self discharge figures in the Consumer article were no doubt measured when the batteries were brand new. While it's interesting to know, this is next to useless information as the self discharge characteristics of non-LSD NiMH batteries can change significantly after even a small amount of use. The self discharge of brand new non-LSD batteries is not characteristic of how they will perform later in their life. The test thus favours the non-LSD batteries. Give them a couple of dozen cycles and then see how well they do for self discharge! This would be a far more realistic test and give far more useful information. The voltage under load should also be measured as it is lower the longer a battery has been sitting idle.
The self discharge test was only 12 weeks long. Longer term testing has shown that brand new fully charged NiMH batteries will still have usable capacity after 3 months (13 weeks), perhaps losing 20-25%. However, after six months it would be very unexpected if any new non-LSD batteries had any capacity left at all – their self discharge rapidly increases after 3 months. Conversely the best LSD batteries would probably not have lost much more capacity – their self discharge slows down after 3 months. Thus, using a comparatively short self discharge test also favours the non-LSD batteries.
It seems disingenuous of Consumer to have tested new batteries after only 12 weeks when any good researcher looking into self discharge should have unearthed this information and been able to devise a fairer, more representative test.
It's also possible that battery manufacturers are just gradually incorporating some of the LSD manufacturing methods in their non-LSD batteries, leading to more robust batteries that (at least when new) self discharge at a slower rate.
Many people think the capacity of a rechargeable battery is the most important characteristic. It's a single number that's easy to understand and allows batteries to be easily compared. Unfortunately the claimed capacity is seldom the actual capacity, especially with non-LSD batteries after they have been used for several cycles. The capacity also makes no consideration of voltage under load, which affects the power the battery can supply; nor does it indicate how reliably the battery will maintain its charge if left for several days or what voltage it will be able to supply under load after an extended storage time.
Whether battery users realise it or not, low self discharge is far more desirable to the average user than high capacity. This is because most people want a reliable battery that will have a good charge when they come to use it. They don't want to worry about high self discharge and unreliable batteries, or having to plan when to charge batteries so they'll be ready just before they're needed. This is one of the main reasons that people don't use rechargeable batteries and why they tend to just sit in a drawer doing nothing.
Low self discharge batteries are what most users should be buying.
Alkaline batteries were recommended by Consumer as "still best for an emergency pack because they lose very little charge when stored for extended periods." It was a strange recommendation because the test was exclusively for rechargeable NiMH batteries; no test data or other supporting evidence was included or even referred to in support of any use of alkaline batteries, and no test data was included regarding LSD batteries after extended storage. Their recommendation was completely unsupported by evidence.
The truth is that alkaline batteries may hold their charge, or they may be unusable after a relatively short period of storage. Alkaline batteries can go flat or even leak in storage, and exposure to heat is particularly bad for contributing to this. Alkalines should be stored in a cool location. Even without heat, the voltage that alkalines can provide under load slowly deteriorates, so alkalines stored for long periods may be rejected by high drain devices (even if they weren't already rejected brand new).
Alkaline batteries do not cope well with moderate to heavy loads, and do not compare well with good NiMH batteries in those situations. Read Roy Lewallen's excellent "1.2 Volt" vs. "1.5 Volt" Batteries PDF for further information.
Quite apart from the question of whether the batteries will be able to do what you want when you eventually need them, the capacity (and stored energy) of alkaline batteries declines over time, so putting alkaline batteries in an emergency kit is basically evaporating your money.
It's a shame Consumer didn't bother assessing how good LSD batteries might actually be in an emergency pack. Some LSD batteries presently on sale are rated for 70% charge after 3 years – easily enough capacity to provide a useable amount of runtime. Consumer only looked at self discharge after 12 weeks. Consumer's claim they are not suitable is not based on their own testing and is an argument from silence.
This is yet another failing of the Consumer test.
Besides having a good shelf life, LSD batteries are of course fully rechargeable at any stage, and usable for any temporary task at any time without worrying about having to buy a new set of batteries to replace them with in the emergency pack. This is a significant improvement in convenience. Rechargeable batteries only have to be used a handful of times before they're more cost effective than alkalines – less than 3 times for the top brands of alkalines.
LSD batteries are now a real option for emergency pack batteries.
FWIW the next (third) version of Eneloop was announced earlier this month (October 2011). They will retain 90% charge after 1 year and 70% charge after 5 years, and claim 1800 (slow) recharges. Initially they'll be sold only in Japan, starting on 14 November 2011.
Update Sep 2013: These are now being sold at PB Technologies.
Update Sep 2016: The generation of Eneloop presently sold by PB Tech claims 65% capacity retention after 5 years and to deliver up to 2100 (slow) recharges, but the latest generation of Eneloop claims 70% capacity remains after 10 years, with up to 2100 charges.
Update October 2017: Consumer was completely wrong about Eneloops being unsuitable for emergency packs. That's not too surprising, since they reached their conclusion completely without (and even ignoring) evidence or adequate research into the low self discharge ability of Eneloops. I've just tested some "brand new" 3rd generation Eneloops which were actually manufactured over four years ago and just sitting in a box (or on a shelf) since then, unopened. When Eneloops are made they're given a ¾ charge, and because they only slowly discharge, they still have a pretty good charge when they're bought and used for the first time. These ones still had 83% of their inital ¾ charge. The evidence is clear: Eneloops are great for emergency packs.