With the Winter Olympics almost upon us, we will be hearing a lot about “perfect” scores in the sports where judges assign points for such things as technical difficulty and artistic interpretation.
A “scoring” system for genealogies would be interesting. If, for example, we had ten categories on which to judge a genealogical source, and each category had a potential ten points maximum, the “perfect” score would be 100. Of course, this would all be subjective, but it would give us a way to group works for comparison (top 10%, bottom 50% etc.). Here are ten categories that I came up with, in no particular order (once we have the categories, we will examine them in detail in future posts):
Author: Does/do the author/authors have credentials and/or a body of work that encourages the reader to believe their product is likely to be reliable (normal human error aside)?
Citations: Is the work supported with fully cited footnotes, endnotes, bibliography?
Completeness: Has the author done an exhaustive search for all pertinent and available records? Does the work go beyond names, dates and places to include deeds, probate, town/colony/state records, military, church, cemetery, etc.?
Format: Is this a genealogy arranged in an accepted format such as Register style or as an Ahnentafel, etc.? Is the information organized and clearly presented?
Scope: Is this a work tracing all descendants of an ancestor (male and female lines), or only male descendants with the surname, and, if so, for how many generations? Is it an “all my ancestors” compilation for one individual, tracing single lines back to particular ancestors?
Would the author have had access to records or family information not available to us now?
Age and Methodology: When was the work compiled? What was the prevailing genealogical methodology of that time? Would the author have had access to records or family information not available to us now? Does the work state information for generations long before the author’s birth without giving sources?
Peer review: Has the work been reviewed, and what were the reviewers’ good and bad points about it? Have there been follow-up articles or subsequent corrections and additions published?
Analysis: Does the author demonstrate the ability to logically analyze and evaluate evidence, compare conflicting sources, discuss problems, document corrections to earlier publications, and suggest further research if indicated?
Restraint: Does the author avoid over-indulging in the imaginary virtues of his/her ancestors?
Access: Is this work accessible? Is it still in print, has it been digitized and made available online, is it available through inter-library loan?
Okay, comments, additions? Next week we will begin with Peer Review.
1. Who’s who
2. Citations
3 and 9. Completeness and restraint
4. Format
5. Scope
7. Peer Review
8. Analysis
10. Accessibility
Great list of categories and am eagerly anticipating the dialogue on such a system. Particularly love category “Restraint” .
Thank you, Sandy.
Marvelous idea. I look forward to further inf. about each topic. Always learn something from your blogs & enjoy reading them. The ones from others also, always learn something. Now if I could only remember ½ of what I read, I would be a better genealogist.
Barbara, If I could only remember half of it, too!
Excellent article. We’ll give it a 9.9! On the subject of “Retraint”, I would think a certain amount of speculation/imagination can add a great deal of colour to a writing, as long as it is carefully prefaced with expressions such as “according to family legends”, or “one would imagine that” etc. Having the latitude for expressing theories, always identified as such, can make the subject much more interesting, and can suggest avenues for further research. On the subject of “Analysis”, one of my favorite aspects is a writer’s ability to place his subject in a historical context, such as relating place names of the time with names we use today, or by mentioning economic or political events of the time which probably drove certain behaviours.
Dave, Imagination is good, but there are some times when authors get carried away. I’ve got some good examples. Historical content is definitely good, but again imagination can get in the way when the author is not well educated — all I have to say is “Mayflower stories”! It will be a fun subject.
I like the the idea and the categories very much, but questions come to my mind, and I’d like to stimulate further comments:
As proposed, each category deserves equal weight. That keeps it simple, but is Author (prior publications) as important as Citations? Is Scope as important as Analysis?
Do some of the categories overlap, thereby giving extra weight? I think you’ve done an admirable job picking the categories but there may be overlap between Methodology and Citations, for example.
I’m probably jumping ahead to your next posts, but what’s a 10 and what’s a 4 could be very different to different reviewers. For examples: (1) Scope, is “all descendants of X” necessarily better than “all ancestors of Y”, especially if the author states her purpose clearly and fulfills that goal? (2) Peer Review, Is an article that has been peer reviewed and corrected necessarily better than one that has been peer reviewed but no corrections issued because the reviewers agreed with the author’s conclusions? (3) Format, would the rater differentiate between Register, Ahnentafel or other standard format or are they all a 10 because they are recognized and in use?
Ed, I hear you (my degree is in Animal Husbandry). It just remains to be seen how the numbers fall out. Ultimately, it may be more useful to report all 10 scores rather than an average or median. Something to while away the winter.
“Restraint: Does the author avoid over-indulging in the imaginary virtues of his/her ancestors?” Alicia, your sense of humor always creeps into even your most serious of posts, and I love it! Especially since all the various other categories make excellent sense to me, but as a rank amateur, I’ll leave future comments to others and just tag along for the ride.
Judy, you are more than rank amateur. You know a lot about what not to believe!
Again you have done an excellent job of outlining the things that matter. I am very much looking forward to seeing the articles as they come out, and the discussions that follow each category.
I believe Alicia is not stating that one of these points has more importance than another. These categories are mentioned as food for thought, that is to say, consideration when one finds a source. Is this source credible? Should I use this source’s information as fact, or as a hint for my own further research and documentation?
It will be interesting to read Alicia’s coming articles and then apply the thoughts to our own work. Each of us has different reasons for family history research — we must seek the truth, but our purposes are different.
Linda, yes, we’re trying for guidelines of some kind. Won’t get me a degree from MIT, I’m sure, but hopefully will be useful.
LOL, ‘restraint’…seriously, most of,us know exactly what you mean, unfortunately. People just need to accept their ancestors for who and what they are…usually the truth is interesting enough.
Carol, it is hard when a lot of the books were written in the “romantic” 19th century and then everyone just keeps copying the same verbiage!
I have long thought that genealogy needed something akin to the evidence grading system used by expert panels developing clinical practice guidelines for medicine. My thinking was narrowly focused on a system for scoring evidence that combines the source evaluation with the information evaluation with the evidence evaluation. I like the idea of scoring genealogies.
Ann, good, I will probably need your help.
Wonderful project. Looking forward to learning from this. Thank you, Alicia.
This sounds like a fabulous inquiry that will help us all see the big picture. I submit one: the disintegration of source from 10 – saw Primary source with own eyes, to: 0 – have a photocopy of compiled vital records of collected records with hand edited margin notes from a second cousin I never met.
It happens all the time, unfortunately.
Alicia, this post alone is a great analysis of what we should be asking ourselves each time we review not just a genealogy, but also family trees, many of the transcribed records we are now seeing in genealogy databases that really don’t have much in the way of a citation, vital records, etc. I hope you will touch on primary and secondary sources as the lack of that knowledge contributes greatly to all that junk genealogy out there. Kudos to you for taking on this epic task and opening a good discussion. (You never choose small, easy projects, do you?? ;D) I look forward to your future posts!
Thanks. My brother says I get into these predicaments because I am a “Williams.” Must be in the DNA.
I would vote for posting all 10 scores. Brings to mind the math lesson about mean,median and mode. A blended score is can sometimes differ from each of the individual points. I hope as your project progressrs, the categories sort themselves into subsets, with some carrying more weight than others. We’re all going to learn a lot. Thanks for doing the heavy lifting, Alicia.
Willetta, thanks. Definitely going to require some work, but hopefully useful.
As others have said, love “restraint.” In her book about her family, my mother-in-law said that her Palatines were “happy and contented;” yet they boarded a small boat in 1710 and sailed to New York. And the only family where college attendance is mentioned is her immediate family; we could not convince her to mention all or none.
Howland, that “all or none” question is common. Things that make more work usually get skipped.
What a great list of categories. I’m really looking forward to the follow up articles and I plan on saving every one of them. With source creditability such a judgment call, it is most advantageous to get input from an experience genealogist such as yourself. Any researcher is tasked with the “burden of proof” but somehow evaluating the credibility of genealogy sources used to substantiate what is believed to be truth seems unusually daunting. I guess that is why I find myself using phrases such as “it is believed at this time,” or “until proven otherwise,” or “current information suggests,” etc. Bottom line, thank you for taking on this topic since, as an amateur, I can use some help with those judgment calls.
Ann, thanks. Will try my best.
If nothing else, this list could serve as a reminder to ourselves on what we need to be on the lookout for as we research. Not sure it should be a scored checklist: as you point our, Alicia, it is pretty subjective, and that does not lend itself well to numerical scores. But a great list to keep in the front of my notebook as I research!
In my own profession, I developed the kind of checklists mentioned above. They were used to determine conditions in field research, where the presence of one condition would set one on a path of questions that would be different than if the condition were not present. All answers were checkmarks, and, like a plant key, ultimately led to some conclusion based on the presence or absence of certain characteristics.
I think we are basically asking ourselves questions like this as we evaluate our sources, and assessing how well each serves to answer our research question. Though the hard part (analysis of a body of research with often conflicting or confusing elements) I do not think can ever be reduced to a score. We can only score the presence or absence of the characteristics of the source itself. And then use that information as we evaluate what the sources cumulatively have to tell us.
I have to say that my favorite is, like others, the “restraint” factor. I am so tired of reading that all my female ancestors were uncomplaining about their lot in life! It is as if a standard boilerplate template were being used for every biography (and I think there probably was, so ubiquitous is that wording). That to me indicates both a lack of insight and empathy! I know from stories passed down that some of those women were anything but uncomplaining, and many were not compliant. Good for them. They passed that resistance down to their descendants, who continued to fight for better conditions and more opportunities for women! My own life and that of my daughters and granddaughters are better as a result.
Sorry, got a bit off topic, but this one pushes a button (in case you didn’t notice!).
Annie, thanks. There are undoubtedly a lot of models out there to explore.
I agree with Annie that if scoring is overly quantified, it might strip out the meaning that comes from making assessments of the evidence. I prefer the checklist approach, too.
Getting too granular when scoring factors may lead researchers into a temptation to rely on the score (abandoning an important feedback loop) when they might be better off re-evaluating the source in line with their more recently acquired evidence and reasoning.
I really like the Perfect 10! Such a great idea, ACW! It can be a very helpful tool. I can’t wait for the further installments.
This is a great discussion! One example that immediately comes to my mind is Charles Banks and “The Planters of the Commonwealth of Massachusetts” (featured on Vita Brevis here: https://vita-brevis.org/2014/11/banks-planters-of-the-commonwealth/). Banks is the go-to guy for the history and early genealogies of Martha’s Vineyard, and would probably get a high grade in the Author category. However, his work seems to have significantly gone downhill by the time this work was published in 1930. Evidently working from abridged passenger lists compiled by Samuel G. Drake 1858-60 and published in the NEHGR shortly thereafter, Banks turned “Nicholas Butler and wife Joice, 3 children & 5 servants” into a list of Nicholas, Joice, 4 named children (Thomas had died several years prior to their migration in 1637!) and 4 servants: John Pope, John Gill, Richard Jenkins, and Simon Athearn. This has caused great confusion amongst Athearn descendants, because according to his gravestone and other records, he was born around 1642. Someone -5 years old would have made a very poor sort of servant! This has led to conjecture that perhaps the generally acknowledged Athearn immigrant was really the son of an older Simon Athearn. As it turns out, the NEHGR in 1921 had published a COMPLETE list of those who accompanied Nicholas and Joice Butler in 1637: their children John, Henry, and Lidia Butler, plus servants John Pope, John Gill, Richard Jenkins, Margaret Angels and Christian Spice. Thankfully it appears that we can put that conundrum to rest now, but it does illustrate how even apparently authoritative sources need to be checked and rechecked.
I would like to know what the qualifications of the Judge would be? Do we have more than one Judge? Do we toss out the high and the low scores? Many of the categories would benefit from a Judge with lots of experience. For example, not knowing the depth of potential resources for discovering evidence might lead one to conclude the search was exhaustive when in fact it was not. Seeing and recognizing a long list of secondary sources may suggest an exhaustive search or just lazy work and that no actual research was conducted.
Michael, This is collaborative, we will all share and compare our opinions.
Ms. Crane, I am looking forward to following this discussion. Your categories are similar to what I learned about in the online Boston University Genealogy course. Genealogy as a hobby has turned into a great learning experience because of articles like this one.
I enjoyed this article immensely. Interestingly, I have an ancestor whose parents are identified by an author of a recently published work that is without any supporting data or citations. I disagree with the author’s findings, but am unable to prove my theory or hers. In the meantime, I see that it has been cited by NEHGS. I wonder what score this published work was awarded.
Finally! Directions toward the mountains of information collected but not assessed by this “genealogist,” with paths to explain why this is right but not that. One has to hurry in the last years of a busy life. Thank you for this textbook!