The Basics of Logical Analysis: Making Judgments

A writer recently expressed doubts to me about making judgments, which is a pretty common reservation: there are good reasons that we don’t want to be overly or inappropriately judgmental.  At the same time, however, life is filled with judgments that we have to make, and we want to make them well.

Life is filled with choices and each choice is a judgment. Are you going to get out of bed now, or will you roll over and pull up the covers? Are you going to go out or stay home? What will you wear? How will you prepare to go out? What will you bring with you? What will you eat? Where will you go? etc. etc.

In positions where experience and/or expertise are required, it is because of the difference between people who can make good judgments and people who make bad ones. You want your doctor/dentist/teacher/lawyer/accountant/auto mechanic/public transit driver/etc. to make good judgments when serving you, for example. You want people making policy, whether for business or organization or government, to make good judgments.  And if you aspire to fill any such role yourself, then you need to be able to make judgments yourself.

Most judgments are complicated, and that’s why people use analysis. The word “analysis” is fraught with a certain mystery or awe—for many, at least—but “analysis” is something that we all do pretty commonly. At its root, the word “analysis” means “to take apart” (in contrast to “synthesis” —put together), and at its root, this is what most forms of analysis do: they start to “take apart” the factors that make up a situation where judgment is required.

Consider a really simple example of analysis that most of us have experienced: you go to a store and there are two similar items that might satisfy your basic needs. Let’s say it’s a food item. Most people will perform an informal analysis that might be quite detailed: they compare prices (dimension 1), sizes (dimension 2), ingredients (dimension 3), as well as, perhaps, reputation of producer (dimension 4), aesthetics of packaging (dimension 5).  We could accurately call that comparison a “multi-dimensional analysis,” and it’s one that people do all the time.

This kind of analysis might continue with many of these factors.  With respect to ingredients, we might say “I’m glad they use X in this product, but I’m allergic to Y.” And then we’re analyzing.  An ingredient list, of course, itemizes different constituent parts of a product, so it’s already an analysis of the product.  But we could do the same with the packaging. Indeed, I said “aesthetics of packaging” above, but that’s only one dimension of an evaluative analysis of packaging—in addition to appearance, we might consider materials (paper vs. plastic, for example, is an aspect of packaging that producers absolutely care about; consumers might not be as concerned) and protection of contents. And protection of contents, might itself have different dimensions—for example, preservation of freshness and preservation of form (a cardboard box, for example, will protect the shape of brittle foods—e.g., chips, cookies—better than a plastic bag, but the plastic bag might preserve freshness better).  And if we started studying preservation of freshness, we might start to see different dimensions, again carrying out analysis. I have not studied preservation of freshness, but in my informal off-the-cuff analysis right here, we might consider freshness over weeks, over months, and over years as being different dimensions of preservation. We can imagine packaging that is inferior in the short run, but superior in the long run. For example, a loaf of bread stored in a paper bag will go stale faster than a loaf stored in a plastic bag, but storing a loaf in plastic can make a loaf’s crust less crunchy, which for some breads is a bad thing. (This basic analysis stems from an analysis of desirable qualities of bread—I like crunchy crusts and I like bread that is not stale.)

We analyze almost automatically: we see a movie and like something about it—“I liked the star; I liked the cinematography; etc.—and we have begun the process of analysis. We go to a restaurant and we analyze: “I loved the food, and the service was great, too!”  However, in situations where there is more formality—in educational settings, or when writing and imaging the response of critics—we don’t think of applying the same basic skills that we would apply automatically in our daily life.

Reasons people hesitate to analyze. 

1. We may not feel qualified. As I’ve described it, analysis is a really basic process that we all do, pretty much all the time. But “analysis” is a term typically associated with high levels of expertise. Things like statistical analysis or psychoanalysis or systems analysis are all tasks for experts.  If you doubt yourself—as many do (cf. imposter syndrome—then it is easy to put the tasks of experts outside your own set of abilities.  But, in fact, these formal systems of analysis are no more than extensions of the basic analysis I have described.  The formal details are an outgrowth of repeated attempts to use analysis productively and the recognition that formal systems of analysis are useful.  But those formal systems of analysis all start with the basic willingness to look at something and respond to the complexity that you see. Psychoanalysis, for example, looks at different components of a person’s psychology—id, ego, and super-ego is one analytical axis; conscious and unconscious is another; identification of distinct life-shaping events is another.  Such formal systems of analysis may be detailed and complex, but their use is acquired through practice that starts with trying to identify different issues of significance.  To be an expert in analysis requires practicing analysis, and that means practicing analysis while not yet an expert.  Our analyses, after all, need not commit us to anything. If we feel that an analysis has not helped us, we are perfectly free to ignore it or redo it as we wish.

2. We may feel it is inappropriate.  There are at least two reasons (in addition to the fear of being unqualified): 1. Analysis is often tied to evaluation and negative criticism, which can lead people to avoid it out of a desire to avoid being judgmental. The unfortunate conflation of analysis and negative criticism places analysis in a negative light that it doesn’t deserve. 2. Analysis can also be overdone: not all analysis is useful. Sometimes analysis can be paralyzing: instead of making a decision, we can get stuck thinking more analysis is necessary. And often analysis will focus our attention on negative aspects that we might not have given much consideration. this is not necessarily bad, but it can be an unnecessary damper on enthusiasm. If you enjoy a movie, for example, does it necessarily help you if you suddenly notice a flaw?  Analysis can take attention away form holistic concerns, too.  But these “problems” with analysis are not so much inherent in analysis as they are inherent in misuse. As with drugs or guns, use need not be misuse. There are valuable uses for drugs, for guns, and for analysis. It lies with the practitioner to use with care. 

Research and Analysis

Analysis comes naturally in research. Every choice of topic starts with separating a focal topic from the rest of the world.  If we study “education”, we’re focusing on one part of the world (and leaving out others); if we study “business,” or “history,” or “biology,” again, we’re choosing to separate one aspect of the world from others.  This is not to say that we need imagine any of these ideas as completely distinct from the rest of the world, but only that for various reasons, we are separating out one thing we want to focus on from others that we do not wish to consider. (Or we might have a more sophisticated analysis that separates the world into three general classes—the focal issue, closely related issues, and issues of little direct relevance).  Choices like this are the basis of research, so you want to make them.  

If we ask “how does X affect Y”, a starting place is to literally break out and examine each piece of that sentence: what is X? what is Y? what kind of effects are you imagining?  That is to say that we look at the sentence and separate out different aspects, with each word representing an aspect of the situation in question. The very language that we use reflects our analytical tendencies. Defining different terms used in research is a fundamental process of analyzing a situation into component parts.

Suppose, for example, that we are looking at Montessori education’s (X) effects on students (Y), then  we would naturally want to explain what Montessori education is and who Montessori students are. We would also want to consider what “affects” means in this context, and with a little thought, we can probably find a number of different things that could be relevant to this discussion: maybe Montessori education affects students’ overall success as students, or maybe it affects their emotional health as children, or their ability to make friends, or their long-term success in school, or their success in college, or their success as students of STEM subjects, or their success as students of language arts.  Each of these possible implications for an educational system on its students is one of the factors identified by this very informal process of analysis that I have undertaken.

Or, suppose that we are interested in management practices (X) and business performance (Y).  First we need to look at what we mean by management practices—what counts as a management practice? And if many things do, will we choose to study all of them?  Then, separately, we can look at different dimensions of business performance, starting, perhaps, with profitability, but also including such things as employee morale.

Research starts with casual analysis

Research depends on analysis in many different forms—from finding different aspects of situations to examine to finding different perspectives from which to analyze a situation.  All of these forms essentially spring from the observations that you have as a researcher and your interest in and attention to detail.

In the course of your research, you will probably be motivated to move beyond the initial steps of casual analysis that you would carry out in everyday life—you don’t need to exhaustively list all the different possible characteristics of a movie to decide whether you want to see it (or whether or why you enjoyed it). But don’t be afraid of those first steps: analysis is not something inappropriate or reserved for some special class of analyst. It is one of the foundations of critical thinking, and if you want to come up with original research, your observations of the world, the way that you organize your observations, and the analyses that you come up with are the roots of original research.

So look closely, don’t be afraid to identify specific details, and then see what you can learn from those observations.  At its root, analysis is something we all do. Research is just a move to try to formalize this common practice.

Standards of Judgement

How do we make a decision? What kinds of evidence and certainty do we use to make a judgement?  And do those standards vary from one context to another?

This discussion came out of a conversation on current events I had with my uncle. I am interested in the underlying principles involved, not in the specific issue, and so I am going to focus on abstractions rather than on specifics.

As an aside: I believe that political discourse should focus on principles, not on partisanship, because partisanship influences decision making, and specifically pulls attention away from crucial factors. It would be best, I believe, if we judged plans on their merits, rather than on who proposed them.

The abstract principles that I want to discuss are related to the different contexts in which different standards of judgement apply.  I don’t have a full general theory, but obviously one wants to apply more stringent standards of judgement for more consequential decisions.

One would reasonably want, for example, more evidence and greater confidence when buying a $500 object than when buying a $5.00 object because the cost of error is 100 times greater (obviously financial impacts are partly measured in relative terms: a billionaire might not give more thought to the $500 than to the $5.00, but the general principle holds even if we have to adjust the numbers).

One would reasonably want, for example, more confidence in the reason behind an active plan (e.g., a healthcare intervention) than in presenting a research conclusion about that active plan because healthcare interventions ought to be supported by the conclusions of several studies. Each individual piece of research has to live up to one standard; implementation of a policy or intervention requires that several pieces of research all live up to that same standard.

One would reasonably want, for example, a higher standard of confidence before convicting an individual to loss of freedom or loss of life than convicting them to a financial penalty. This distinction is held in U.S. courts, where criminal cases are held to a standard of being beyond a reasonable doubt, but civil cases can be decided by the preponderance of evidence.  This legal distinction is the one that is of direct concern to the current event that interested me and my uncle.

Let’s talk generally about the situation: a man is accused of a crime. What standards of evidence do we want to apply to this man? Obviously, I’m going to argue that this is contextually sensitive.  If we are talking about getting a conviction in a criminal court, then the standard of judgement is clear: the guilt of the accused must be determined beyond a reasonable doubt.  On the principle that “it is better that ten guilty persons escape than that one innocent suffer” (Blackstone’s formulation), we want to be absolutely certain that the accused is actually guilty.  It’s not enough to have a credible accuser and circumstantial evidence. The burden of proof is very high when we are speaking of meting out punishment.

If we were speaking of rendering a conviction of this man in a civil case, where the penalties are generally financial, a lower standard of proof would be required in U.S. courts—a  preponderance of evidence, which is could be interpreted as saying that guilt is more likely than innocence, even if there is significant doubt.

What about some other situations in which someone might render judgement on this man? What if the accused asks a person out on a date? What standard of confidence does that person need to apply? I would argue that she/he doesn’t need anything close to a preponderance of evidence. All a person needs to reject a potential suitor is the slightest reservation.

What about a business considering hiring the accused? What standard of judgement should that company apply?  There may be legal statutes that restrict businesses’ choices, but on the whole, businesses hiring individuals should should have some discretion to reject a candidate about whom they have doubts, even if there is less than a preponderance of evidence.  The situation is inverted because now the question is about presenting an award, not about meting out punishment. Indeed, I would go so far as to say that it might be reasonable for a company to assume something like the “beyond a reasonable doubt” standard, but in reverse: the company might reasonably say “We will not hire the accused unless his innocence can be proved beyond a reasonable doubt.”  That’s pretty stringent, but a company is looking for the best performer, and if there is any reasonable doubt about a person’s guilt—even if there is a preponderance of evidence suggesting innocence—that might be a good reason to choose an alternate applicant.  

In that last phrase, I added a consideration that has not been previously mentioned, but that is certainly relevant in many situations.  Sometimes, the standard of judgement is not absolute, but is relative, and the focus is no longer on the individual accused but the organization that is looking for someone to fill a position. For a desirable job, there might be several applicants. In such a situation, the standard of judgement with respect to a person accused of a crime is reduced due to the presence of other job applicants: if the accused individual is no more qualified than some other person who is not accused of any crimes, a company might well decide to choose the person who has never been accused over the person who has been accused with no evidence beyond the accusation. The company might be very concerned to maintain the image of probity that is supported by choosing an applicant who has not been accused of any crime.

And if the job in question is a seat on the Supreme Court of the U.S., what standards should we apply? If there are a list of several candidates, amongst whom you are otherwise indifferent (Leonard Leo of the Federalist society said that the Federalist Society had a list of many potential candidates to fill the seat vacated by Justice Kennedy, and the “The list is really good…You can throw a dart at that list and in my view you would be fine.”), what standard needs to be applied to a candidate accused of a crime?  Can a candidate be rejected because of an unproven accusation? 

On the principles that I have discussed, the “beyond a reasonable doubt” standard is not the appropriate standard. A candidate accused of a crime need not be proven guilty beyond a reasonable doubt, as would be required for a criminal conviction. A candidate accused of a crime might reasonably be passed over even if the accusation is unfounded because, as noted before, the question is not of punishment but of reward, and because there are several equally qualified candidates who do not stand accused of a crime.