Irish
10-17-09, 17:35
I really enjoyed reading this article and thought I would post it here as it is relevant to many discussions on this forum as well. Please feel free to comment. http://www.zaphaudio.com/evaluation.html
Subjectivity vs objectivity
subjectivity
noun: judgment based on individual personal impressions and feelings and opinions rather than external facts
objectivity
noun: judgment based on observable phenomena and uninfluenced by emotions or personal prejudices
For those who are not familiar with the usage of these terms in the audio world, subjectivity is an interpetation of performance using ears and music while objectivity is proof of performance using as many scientific measurements and methods as required to make a point.
This is going to be the part of this article that hurts some the most: In a situation where proven methodology and technology exists to do objective testing, people will side with subjectivity when 1)objectivity does not give them the answers they want to hear 2)they have an agenda or interest to protect, or 3)they are incapable of executing objective methods or reading objective results. Why do I never discuss this issue on the forums? Because there's no winning an argument with someone who is stubborn and uneducated, unaware, or uninformed. (ouch, but I can't think of any less harsh terms at the moment)
Don't let a proponent of subjectivity tell you that the right measurements don't exist to properly describe performance. That is a wrong statement, occasionally stated unknowingly but more often stated blatantly on purpose. The right measurements do exist and it's time for people to get with the program. The field of audio is very well known and has been for 50 years. My advice to those who say the right measurements don't exist: Get a decent measurement package, start measuring some drivers, and then start listening to them, in different applications and without filters, alone and in systems with other drivers, just so you can hear exactly what you see. In time, understanding will come.
Beware the subjective speaker review. Anything subjective is likely full of vague useless verbage and is open to very different interpretation by anyone. It's also open to omissions because the review material will always be limited. And, as mentioned above, there's a good chance it's just not true.
A speaker can't be evaluated like a painting. The nature of a painting forces subjectivity while a speaker *can* be objectively tested. Don't forget that a painting is a production conveying the emotions and ideas of the painter, while a speaker is a device for reproduction. There is no room for emotion (or distortion) in a reproduction. A reproduction is judged by one thing: accuracy compared to the original. People shouldn't put on purple sunglasses when going to an art museum.
The objectivity of measurements leaves nothing to the imagination. Issues can not hide or be neglected, intentionally or unintentionally. So obviously, the moral of this story is that objective measurements should be used whenever possible, and subjective reviews should not be accepted or at least considered with a grain of salt.
How often has someone said "Forget the measurements and tell me how it sounds." Or "Learn to listen with your ears." Well I hate to say it, but stubborn and uninformed is rearing it's ugly head again. A lack of understanding without an open mind leads to a refusal of the facts and an embrace of whatever view happens to further that individual's cause. Or worse yet, it may lead to a view that's simply an attempt to make a knowingly guilty person appear smarter or more interesting than they actually are.
All the answers are in the measurements. And I mean ALL the answers. Some people don't know how to interpet the measurements. Some don't want to know all the answers. The mysticism of not knowing how a speaker sounds before trying to use it might make for an interesting, though lengthy design session. That's fine. If that's the route chosen, so be it and enjoy the design process. It would be wrong however to say that's the only route to take. The sad part is that the less technical "guessing" route is the one that's more likely to lead to failure.
The performance of a driver can simply be broken down into two types of distortion - linear and nonlinear. Everything is a subset of that. It's beyond the scope of this article to describe how types of distortion are categorized. As far as the measurements posted on this web site, don't assume that if one type if distortion isn't tested for, something is missing. The linear distortion in the form of frequency response, and non-linear distortion in the form of harmonic distortion and intermodulation distortion are the factors that make up 95% of a driver's performance. For example, I don't show "the rise and fall time" of a driver because it's just a derivative of the response curve. It's all in the impulse.
Ears cannot be trusted
Those who don't measure, don't know. Period. They can pretend they know, but rest assured they don't. The human ear, as it is connected to the human brain, is not very smart and easily fooled. If something is wrong with a speaker, people have a hard time telling what it is. Using ears only, the quality of a speaker can only be described in extremely subjective terms. Treble is often described as bright, dull, edgy, recessed, etc. Bass is often described as warm, boomy, deep, lean, tight, etc. But without any accurate basis of comparison, comments like that are meaningless. Don't even get me started by describing how many "veils have been lifted" from the music or how a speaker's "pace rhythm and timing" is affecting the sound. Those vague, meaningless statements are made by people who lack the proper technical vocabulary to describe a speaker's performance. In summary, a driver should be fully measured before a person is qualified to comment on the sound of that driver.
Look at it this way: who is more believable, the guy who says "This tweeter sounds edgy" or the guy who says "This tweeter has moderate 3rd and high 5th order harmonic distortion." Again, who is more believable: the guy who says "This tweeter sounds dull" or the guy who says "The average level of this tweeter is 1.5 dB lower than the woofer above the crossover point." So, while "this tweeter sounds dull" does imply that something might be wrong, there is absolutely nothing there to say what is wrong or even offer any proof that something actually is wrong.
To make a point, some statements could have many different translations. Here's a bunch of examples. We'll start with a single statement made by folks that have been listening to crap.
This tweeter sounds dull. Translation: "I've been listening to a speaker without baffle step compensation for 10 years, and this new design sounds different than my personal standard."
This tweeter sounds dull. Translation: "My last system had Dynaudio D21 tweeters, and now anything without a peak at 10kHz sounds recessed and doesn't have sizzle I need."
This tweeter sounds dull. Translation: "My current speaker has a woofer that did not have the breakup node properly filtered, and now I've accepted that type of sound as normal."
Then we'll try the statements made by folks with well trained ears but other problems.
This tweeter sounds dull. Translation: "I misunderstood the crossover diagram, hooked the tweeter up with the polarity reversed and now I have a broad 40dB null at 2.5kHz."
This tweeter sounds dull. Translation: "My living room is very sound absorptive, and any system with a flat response just isn't enough to overcome the lack of room reflection."
This tweeter sounds dull. Translation: "Oops, my kid pulled the treble knob off my preamp, put it back rotated 45 degrees to the right, and then adjusted it so it visually centered."
Go ahead openly laugh at that last one... but it has happened. Here we have addressed about 1% of the potential reasons why a tweeter may sound dull, and we have not even considered that there may be a problem with the design or an out of spec driver. And good lord, "this tweeter sounds bright" has a potential 500% more translations. Obviously, words can't cut it. A picture may be worth a thousand words, but in a case like this, an actual measurement is priceless.
All that said, it's not impossible to speak in terms of distortion types when listening with ears. But be aware that it takes years of working with real distortion measurements before types of distortion at various frequencies can be properly identified using only ears. Enough said.
Drivers cannot be evaluated in a system
I see this all the time. Statements like "I used XXX driver in my last YYY design and it sounded ZZZ." Replace X, Y and Z with anything. It doesn't matter what, because statements like that don't have any merit. When a driver is used in a system, only the system as a whole can be evaluated. (and even then, as mentioned above, not without measurements) Of course the drivers are making sound, but what you really hear is the crossover, design choices and the listening environment. So many people evaluate drivers while they are in a poorly designed system and then blame the result on the drivers themselves. Only individual and extensive driver measurements are acceptable in evaluating an individual driver.
If distortion measurements aren't part of the first step in selecting and using a driver, the system "tweaking time" will multiply exponentially as the designer struggles to find a configuration that helps hide a driver's faults. All drivers have faults. Without distortion measurements, improper usage of a driver may cause a person to simply give up and come to the wrong conclusion that a driver is a poor performer. In a case like that, the only poor performer is the designer who failed to work around a driver's faults.
Where subjectivity actually works
During the driver evaluation process, any sort of subjectivity is a bad idea. But on the other hand, subjectivity during the system evaluation process is going to be required. At some point, a designer has to decide on tradeoffs. That could mean deciding what types of distortion are more annoying to the designer personally. For example, what sounds worse: a broad, large increase in even order harmonic distortion in the lower midrange and bass or sharper but narrow band odd order harmonic distortion in the upper treble? Sound familiar? It's the old metal vs paper cone debate and it comes down to personal preference. The choice is a subjective one. Some prefer metal cones while many others prefer paper or poly. That's OK. Measurements can tell the truth about a driver but remember that they can't tell you what you like.
When a proponent of subjectivity repeats what I just mentioned above, I'm not going to dispute it. Many things in the end come down to personal preference. A system design decision can be subjective, but testing and evaluating drivers individually must remain objective.
Detail is a crappy word
This is a word that is used by people who don't know how to properly describe a speaker's performance. It is merely a vague indicator that something sounds pleasing, and it means different things to many different people. It could mean more distortion or it could mean less. Detail is an audiophile "power word" that means nothing but sounds good rolling off the tongue. It sure sounds better than just mumbling "I like it" every time a speaker impresses.
I might advise substituting two different words that work a little better: clean and smooth. Clean can mean low non-linear distortion. It would mean a lack of harmonic frequencies that are different from the fundamental frequency reproduced. Clean can mean hearing only what's in the original music and nothing more. Smooth can mean low linear distortion. Of course this could mean a smooth frequency response, but it also means low energy storage which is directly related. Rise and decay time are all related to frequency response smoothness.
The crappy words and phrases list
Aside from the word detail, there are a lot of other audiophile wannabe "power words" and phrases. All are vague, non-descript and useless. Some phrases are deceiving, such as soundstage width, depth, or height. What people are really hearing is a system's power response and it's interaction with the room. Open up a Stereophile and read a page to obtain a plethora useless audiophile terms. Some of the offending single-words might be transparency, image, bite, snap, grain or resolution (in reference to anything analog) and so on.
Multi-word phrases are even worse. Veils have been lifted gets a special achievement award for overused bad cliche. It doesn't end with phrases like pace, rhythm and timing and black spaces between the notes. Any description of music played during evaluation or any description of cables or interconnects used during evaluation also fall under this category. I won't go on for fear that I might cause nausea. Just be aware that there's a lot of it out there. Some of it is obvious and some is not.
Common system comparison errors
Non-linear confused with linear distortion
For those who are not familiar with the terms, non-linear distortion is any type that adds additional frequencies to the original fundamental frquency. Linear distortion is anything related to deviation from flat frequency response. Harmonic distortion by it's nature can make a speaker seem to have a bright tonality. It might measure flat, but upper harmonics not in the original recording can change perception of the tonality. It takes a deep familiarity with a reference recording and a keen ear to be able to tell the difference between linear and non-linear distortion.
Woofer issues confused with tweeter performance
Related to the above, a poor choice of a crossover point on a woofer (too high) can often lead a person to believe there is a distortion or level problem with a tweeter. A woofer could be generating harmonics well into the range that the tweeter is covering. When a listener declares that they don't like a tweeter they heard in a system, 90% of the time it's the woofer and the design that's actually the problem.
Low end distortion is not good bass
Smallish woofers often trick people into thinking bass performance is good. There's a mindset that more is better, but when listening to tiny woofers trying to reproduce the bottom three octaves, you're not hearing "more" bass, you're hearing "different" bass. Bass distortion causes audible harmonics above the fundamental. On a small scale, this sounds like a warm tonality. On a large scale it can't be any better described than aural mud. Be carefull before declaring a woofer as having good bass. If a person is unfamiliar with what good bass is, they could just be hearing muddy distortion.
Dull or veiled vs vibrant or exciting
When comparing the sound of two systems side by side, the first impression is often that the more distorted speaker seems more detailed, and the more accurate one seems dull or veiled. A speaker could often be described as vibrant or exciting by ears that are unfamiliar with clean sound. Even when set to the same level, the more distorted speaker will seem louder, and louder immediately seems better. Eventually, the distortion that causes "vibrant and exciting" will turn to listening fatigue, but it happens slowly. It's usually too slow to make quick A-B comparisons useful. If you really want to compare speakers with your ears, you had better live with them for a good long time. Otherwise, take some measurements to get the truth right away.
The fact that listening fatigue takes a long time to set in does not bode well for making comparisons on a showroom floor, or at a DIY event for that matter. The favorite of a DIY event is most likely not the best performing speaker. On the showroom floor, it's a well known fact that some speakers are intentionally inaccurate to help them stand out. The unknowing consumer then takes home a poor performing speaker only to realize days or weeks later that this speaker doesn't sound as good as they thought. The DIY'er building a project they heard at an event is often disappointed with the result after living with it a while.
Subjectivity vs objectivity
subjectivity
noun: judgment based on individual personal impressions and feelings and opinions rather than external facts
objectivity
noun: judgment based on observable phenomena and uninfluenced by emotions or personal prejudices
For those who are not familiar with the usage of these terms in the audio world, subjectivity is an interpetation of performance using ears and music while objectivity is proof of performance using as many scientific measurements and methods as required to make a point.
This is going to be the part of this article that hurts some the most: In a situation where proven methodology and technology exists to do objective testing, people will side with subjectivity when 1)objectivity does not give them the answers they want to hear 2)they have an agenda or interest to protect, or 3)they are incapable of executing objective methods or reading objective results. Why do I never discuss this issue on the forums? Because there's no winning an argument with someone who is stubborn and uneducated, unaware, or uninformed. (ouch, but I can't think of any less harsh terms at the moment)
Don't let a proponent of subjectivity tell you that the right measurements don't exist to properly describe performance. That is a wrong statement, occasionally stated unknowingly but more often stated blatantly on purpose. The right measurements do exist and it's time for people to get with the program. The field of audio is very well known and has been for 50 years. My advice to those who say the right measurements don't exist: Get a decent measurement package, start measuring some drivers, and then start listening to them, in different applications and without filters, alone and in systems with other drivers, just so you can hear exactly what you see. In time, understanding will come.
Beware the subjective speaker review. Anything subjective is likely full of vague useless verbage and is open to very different interpretation by anyone. It's also open to omissions because the review material will always be limited. And, as mentioned above, there's a good chance it's just not true.
A speaker can't be evaluated like a painting. The nature of a painting forces subjectivity while a speaker *can* be objectively tested. Don't forget that a painting is a production conveying the emotions and ideas of the painter, while a speaker is a device for reproduction. There is no room for emotion (or distortion) in a reproduction. A reproduction is judged by one thing: accuracy compared to the original. People shouldn't put on purple sunglasses when going to an art museum.
The objectivity of measurements leaves nothing to the imagination. Issues can not hide or be neglected, intentionally or unintentionally. So obviously, the moral of this story is that objective measurements should be used whenever possible, and subjective reviews should not be accepted or at least considered with a grain of salt.
How often has someone said "Forget the measurements and tell me how it sounds." Or "Learn to listen with your ears." Well I hate to say it, but stubborn and uninformed is rearing it's ugly head again. A lack of understanding without an open mind leads to a refusal of the facts and an embrace of whatever view happens to further that individual's cause. Or worse yet, it may lead to a view that's simply an attempt to make a knowingly guilty person appear smarter or more interesting than they actually are.
All the answers are in the measurements. And I mean ALL the answers. Some people don't know how to interpet the measurements. Some don't want to know all the answers. The mysticism of not knowing how a speaker sounds before trying to use it might make for an interesting, though lengthy design session. That's fine. If that's the route chosen, so be it and enjoy the design process. It would be wrong however to say that's the only route to take. The sad part is that the less technical "guessing" route is the one that's more likely to lead to failure.
The performance of a driver can simply be broken down into two types of distortion - linear and nonlinear. Everything is a subset of that. It's beyond the scope of this article to describe how types of distortion are categorized. As far as the measurements posted on this web site, don't assume that if one type if distortion isn't tested for, something is missing. The linear distortion in the form of frequency response, and non-linear distortion in the form of harmonic distortion and intermodulation distortion are the factors that make up 95% of a driver's performance. For example, I don't show "the rise and fall time" of a driver because it's just a derivative of the response curve. It's all in the impulse.
Ears cannot be trusted
Those who don't measure, don't know. Period. They can pretend they know, but rest assured they don't. The human ear, as it is connected to the human brain, is not very smart and easily fooled. If something is wrong with a speaker, people have a hard time telling what it is. Using ears only, the quality of a speaker can only be described in extremely subjective terms. Treble is often described as bright, dull, edgy, recessed, etc. Bass is often described as warm, boomy, deep, lean, tight, etc. But without any accurate basis of comparison, comments like that are meaningless. Don't even get me started by describing how many "veils have been lifted" from the music or how a speaker's "pace rhythm and timing" is affecting the sound. Those vague, meaningless statements are made by people who lack the proper technical vocabulary to describe a speaker's performance. In summary, a driver should be fully measured before a person is qualified to comment on the sound of that driver.
Look at it this way: who is more believable, the guy who says "This tweeter sounds edgy" or the guy who says "This tweeter has moderate 3rd and high 5th order harmonic distortion." Again, who is more believable: the guy who says "This tweeter sounds dull" or the guy who says "The average level of this tweeter is 1.5 dB lower than the woofer above the crossover point." So, while "this tweeter sounds dull" does imply that something might be wrong, there is absolutely nothing there to say what is wrong or even offer any proof that something actually is wrong.
To make a point, some statements could have many different translations. Here's a bunch of examples. We'll start with a single statement made by folks that have been listening to crap.
This tweeter sounds dull. Translation: "I've been listening to a speaker without baffle step compensation for 10 years, and this new design sounds different than my personal standard."
This tweeter sounds dull. Translation: "My last system had Dynaudio D21 tweeters, and now anything without a peak at 10kHz sounds recessed and doesn't have sizzle I need."
This tweeter sounds dull. Translation: "My current speaker has a woofer that did not have the breakup node properly filtered, and now I've accepted that type of sound as normal."
Then we'll try the statements made by folks with well trained ears but other problems.
This tweeter sounds dull. Translation: "I misunderstood the crossover diagram, hooked the tweeter up with the polarity reversed and now I have a broad 40dB null at 2.5kHz."
This tweeter sounds dull. Translation: "My living room is very sound absorptive, and any system with a flat response just isn't enough to overcome the lack of room reflection."
This tweeter sounds dull. Translation: "Oops, my kid pulled the treble knob off my preamp, put it back rotated 45 degrees to the right, and then adjusted it so it visually centered."
Go ahead openly laugh at that last one... but it has happened. Here we have addressed about 1% of the potential reasons why a tweeter may sound dull, and we have not even considered that there may be a problem with the design or an out of spec driver. And good lord, "this tweeter sounds bright" has a potential 500% more translations. Obviously, words can't cut it. A picture may be worth a thousand words, but in a case like this, an actual measurement is priceless.
All that said, it's not impossible to speak in terms of distortion types when listening with ears. But be aware that it takes years of working with real distortion measurements before types of distortion at various frequencies can be properly identified using only ears. Enough said.
Drivers cannot be evaluated in a system
I see this all the time. Statements like "I used XXX driver in my last YYY design and it sounded ZZZ." Replace X, Y and Z with anything. It doesn't matter what, because statements like that don't have any merit. When a driver is used in a system, only the system as a whole can be evaluated. (and even then, as mentioned above, not without measurements) Of course the drivers are making sound, but what you really hear is the crossover, design choices and the listening environment. So many people evaluate drivers while they are in a poorly designed system and then blame the result on the drivers themselves. Only individual and extensive driver measurements are acceptable in evaluating an individual driver.
If distortion measurements aren't part of the first step in selecting and using a driver, the system "tweaking time" will multiply exponentially as the designer struggles to find a configuration that helps hide a driver's faults. All drivers have faults. Without distortion measurements, improper usage of a driver may cause a person to simply give up and come to the wrong conclusion that a driver is a poor performer. In a case like that, the only poor performer is the designer who failed to work around a driver's faults.
Where subjectivity actually works
During the driver evaluation process, any sort of subjectivity is a bad idea. But on the other hand, subjectivity during the system evaluation process is going to be required. At some point, a designer has to decide on tradeoffs. That could mean deciding what types of distortion are more annoying to the designer personally. For example, what sounds worse: a broad, large increase in even order harmonic distortion in the lower midrange and bass or sharper but narrow band odd order harmonic distortion in the upper treble? Sound familiar? It's the old metal vs paper cone debate and it comes down to personal preference. The choice is a subjective one. Some prefer metal cones while many others prefer paper or poly. That's OK. Measurements can tell the truth about a driver but remember that they can't tell you what you like.
When a proponent of subjectivity repeats what I just mentioned above, I'm not going to dispute it. Many things in the end come down to personal preference. A system design decision can be subjective, but testing and evaluating drivers individually must remain objective.
Detail is a crappy word
This is a word that is used by people who don't know how to properly describe a speaker's performance. It is merely a vague indicator that something sounds pleasing, and it means different things to many different people. It could mean more distortion or it could mean less. Detail is an audiophile "power word" that means nothing but sounds good rolling off the tongue. It sure sounds better than just mumbling "I like it" every time a speaker impresses.
I might advise substituting two different words that work a little better: clean and smooth. Clean can mean low non-linear distortion. It would mean a lack of harmonic frequencies that are different from the fundamental frequency reproduced. Clean can mean hearing only what's in the original music and nothing more. Smooth can mean low linear distortion. Of course this could mean a smooth frequency response, but it also means low energy storage which is directly related. Rise and decay time are all related to frequency response smoothness.
The crappy words and phrases list
Aside from the word detail, there are a lot of other audiophile wannabe "power words" and phrases. All are vague, non-descript and useless. Some phrases are deceiving, such as soundstage width, depth, or height. What people are really hearing is a system's power response and it's interaction with the room. Open up a Stereophile and read a page to obtain a plethora useless audiophile terms. Some of the offending single-words might be transparency, image, bite, snap, grain or resolution (in reference to anything analog) and so on.
Multi-word phrases are even worse. Veils have been lifted gets a special achievement award for overused bad cliche. It doesn't end with phrases like pace, rhythm and timing and black spaces between the notes. Any description of music played during evaluation or any description of cables or interconnects used during evaluation also fall under this category. I won't go on for fear that I might cause nausea. Just be aware that there's a lot of it out there. Some of it is obvious and some is not.
Common system comparison errors
Non-linear confused with linear distortion
For those who are not familiar with the terms, non-linear distortion is any type that adds additional frequencies to the original fundamental frquency. Linear distortion is anything related to deviation from flat frequency response. Harmonic distortion by it's nature can make a speaker seem to have a bright tonality. It might measure flat, but upper harmonics not in the original recording can change perception of the tonality. It takes a deep familiarity with a reference recording and a keen ear to be able to tell the difference between linear and non-linear distortion.
Woofer issues confused with tweeter performance
Related to the above, a poor choice of a crossover point on a woofer (too high) can often lead a person to believe there is a distortion or level problem with a tweeter. A woofer could be generating harmonics well into the range that the tweeter is covering. When a listener declares that they don't like a tweeter they heard in a system, 90% of the time it's the woofer and the design that's actually the problem.
Low end distortion is not good bass
Smallish woofers often trick people into thinking bass performance is good. There's a mindset that more is better, but when listening to tiny woofers trying to reproduce the bottom three octaves, you're not hearing "more" bass, you're hearing "different" bass. Bass distortion causes audible harmonics above the fundamental. On a small scale, this sounds like a warm tonality. On a large scale it can't be any better described than aural mud. Be carefull before declaring a woofer as having good bass. If a person is unfamiliar with what good bass is, they could just be hearing muddy distortion.
Dull or veiled vs vibrant or exciting
When comparing the sound of two systems side by side, the first impression is often that the more distorted speaker seems more detailed, and the more accurate one seems dull or veiled. A speaker could often be described as vibrant or exciting by ears that are unfamiliar with clean sound. Even when set to the same level, the more distorted speaker will seem louder, and louder immediately seems better. Eventually, the distortion that causes "vibrant and exciting" will turn to listening fatigue, but it happens slowly. It's usually too slow to make quick A-B comparisons useful. If you really want to compare speakers with your ears, you had better live with them for a good long time. Otherwise, take some measurements to get the truth right away.
The fact that listening fatigue takes a long time to set in does not bode well for making comparisons on a showroom floor, or at a DIY event for that matter. The favorite of a DIY event is most likely not the best performing speaker. On the showroom floor, it's a well known fact that some speakers are intentionally inaccurate to help them stand out. The unknowing consumer then takes home a poor performing speaker only to realize days or weeks later that this speaker doesn't sound as good as they thought. The DIY'er building a project they heard at an event is often disappointed with the result after living with it a while.