1. Lost in Genealogy: Seven Steps to Battling Bias

    1 Comment

    This excellent article was written by former Pharos student Dr Sophie Kay. She has kindly allowed us to reproduce it here. Sophie is a professional genealogist at Khronicle. You can find Sophie’s blog here: The Parchment Rustler and follow her on Twitter: @ScientistSoph.

    Today, we’re going to talk about the elephant in every genealogist’s research room. It’s one we’ve all spent some time with, whether we realise it or not. And what’s more, this particular elephant tends to divert our research when it shouldn’t. At its worst, it can stampede us right off course.

    Have you guessed the elephant’s name yet…?

    Yes, bias. Bias comes in many forms, but I’m going to focus on cognitive bias here. This occurs when our internal judgments impede rational thought and affect our decision making when we’re interpreting information. So when does this happen and how can we combat it?

    Our Research Process

    Most of the time, there is more than one route through a genealogy problem. Each researcher will tackle things in their own way, with a variety of creative approaches often possible. 

    Whatever the route, the fundamentals of the process involve us examining multiple sources. For each of these, we identify what information that document has given us, and assess how much we can trust it

    This image has an empty alt attribute; its file name is Document1.jpg
    With a multitude of documents in our work, how do we decide which evidence to trust the most? Image credit: Militia return by Liz West, CC-BY 2.0; Birth certificate for Albert Cook by Jez Levy, CC-BY 2.0.

    So every time you look at a document, you’re making a value judgment about its usefulness to determine how it affects the emerging story. These judgments are key to pushing our research forward: they’re a natural part of what we do. Usually, there is no “perfect” way to navigate our decisions either…and sometimes our forebears find ways to surprise us (NOTE: For more about proof, I highly recommend Phil Isherwood’s article).

    But what happens when we make errors of judgment? Perhaps we trust one source more than we should; perhaps we unfairly reject another because we think it doesn’t fit the picture we have in mind. This is where cognitive bias comes into play.

    Cognitive bias takes many forms, but I’m going to focus on two sorts here: confirmation bias, and anchoring. Let’s take a look now at what these are and what they might mean for our genealogy research.

    Confirmation Bias

    Confirmation bias arises when our firm beliefs about a person or situation cause us to dismiss evidence which conflicts with those beliefs. This prevents us from making an objective assessment of the evidence. Instead, confirmation bias causes us to seek out information that reinforces our existing beliefs.

    Baptism register entry for a Francis Burdett Nuttall, son of Joseph and Jane Nuttall of Hines, Lancs. This is shown as a photo of the original document.
    Baptism record for Francis Burdett Nuttall, 18 Sept 1861 (indicated by the red arrow). Source: Ancestry – Manchester, England, Non-Conformist Births and Baptisms, 1758-1912.

    Take a look at the baptism record above and imagine that you really wanted to discover high-born ancestry, despite having any evidence to suggest this. If you found Francis Burdett Nuttall in your family tree, you might jump at the prospect of him being related to – or even descended from – the line of Burdett baronets. Looking at this through the lens of confirmation bias, you might conclude that the similarity of name was “proof” of a blood connection, rather than exploring other possibilities – such as the parents naming their son after a public figure they had admired.

    Confirmation bias can be a particularly potent distraction in genealogy research. To research the lives of our ancestors, we step from the known into the unknown: throughout this process, we have to generate our own theories which govern what to look for next. We all need ideas to follow up, but we need to be careful that we’re motivated by the evidence in front of us, rather than wanting a particular outcome.

    Anchoring

    Anchoring bias occurs when you give greater weighting to the first pieces of information gleaned in your research, whether or not they are of good quality. If we happen upon a misleading piece of information early on, it could easily steer us off course.

    For example, think about research using newspapers, where the details included may not always be accurate. If a news article is your first source of a particular “fact”, it has considerable power to lead you astray. Take, for instance, the following news article and imagine that we came to this early in our research, knowing John Walton’s name but little else:

    News article summarising a Juvenile Plundering case in Birkenhead in 1865, discussing a boy John Walton "about 13 years of age".
    Liverpool Mercury, 17 February 1865, page 8. Birkenhead Police Court summary. Source: British Newspaper Archive via FindMyPast.

    How might we use this article in our research? We cannot guarantee that it provides a fair representation of the facts. 

    Anchoring might occur if you assumed from this that John was precisely 13 at the time of his trial and refused to consider any alternative ages when running your searches. Anchoring bias can make us blind to other possibilities, perhaps cause us to run too-narrow searches or to discount genuine matches that don’t fit with our early evidence.

    How can I battle these biases in my own genealogy research?

    It’s not always a straightforward battle, but the following suggestions may help you break out of the bias bubble:

    1. Slow down and acknowledge the existence of cognitive bias

    This is the absolute first stop on our journey. Each of us, no matter what our level of experience, can fall victim to cognitive bias. By acknowledging this, we are better placed to combat it. Our biases are rarely conscious ones and can be annoyingly difficult to spot. 

    If we’re to call out our biases, then slowing down our pace and questioning our own decisions can be of huge benefit. When we’re hurtling through our research, hot on the trail of a new lead (we’ve all been there), it’s easy to fall into quick decision making without weighing all the evidence. Snap decisions often rely more on “gut feeling” and might allow our biases to sneak in unannounced.

    Once we’ve realised that bias is a universal experience, there are some routes to spotting it, which we’ll delve into now.

    2. Talk to others about your work.

    Talking about your process, not just your findings, is one of the most important aspects of beating cognitive bias. Bias thrives on our isolation. Genealogy research frequently occurs within our own little bubble and it’s typically the end findings that we discuss with our nearest and dearest, not the methods or reasoning we used to get there.

    Researching alone – or even in a pair or small group where you may all get stuck in a thinking rut – can entrench our habits and attitudes and cause us to miss things. Sometimes, a fresh pair of eyes is what’s needed. Think of it as genealogical peer review.

    Two men are having an animated conversation whilst sitting at a table. One of the men is gesticulating with his hands to make a point.
    Engaging with others about our genealogy research – our thought processes, as well as the story itself, can help our findings to flourish. Photo by Daniel, CC BY-ND 2.0.

    So, whom should you speak to? Your confidant could include a trusted friend or relative who has experience in genealogy. Alternatively, you could join your local family history society to meet others with whom you can discuss your research. If you’re UK-based, you might find the Federation of Family History’s society search facility useful. 

    You might also choose to connect with other researchers via online forums and social media. Draw on these valuable community networks and be prepared to reciprocate too: if someone acts as a sounding board for your research, how about you return the favour for them when they need it?

    3. Write up your research.

    When we write up, we draw the separate strands of our research together into a common narrative thread. This is a critical process and requires you to scrutinise your own ideas and methods, and work closely with your notes and sources. This is known as self-reflective practice

    Transforming your research notes into a written volume, an article or a blogpost is a crucial phase of self-reflection, forcing us to critique our own work. Photo credits: Pencil notes by Robert of Fairfax, CC BY-NC 2.0; bookshelves by Eltpics, CC BY-NC 2.0.

    Writing up is a great opportunity to spot any errors you might have made. Think of it as a friendly critique of your younger self! 

    Sometimes a project might need time for it to grow to a point where writing up is a viable prospect. Where possible, I’d advise writing up as often as is appropriate, so you can spot any mistakes or missed avenues before you build on your research much further: backtracking then will feel even more painful.

    4. Embrace evidence.

    An empirical (evidence-based) approach can help you to construct a logical argument for why you think the record you’ve found is the right one for the individual you’re researching. Always ask: what’s the evidence for or against this finding? Does it fit with what I already know, or do I need to rethink some aspects of my existing narrative?

    When drawing together multiple sources from several research sessions to make your case, a well-organised system of recording in a research log can assist you in referring to all the relevant documents when you’re drawing your conclusions.

    Example of a handwritten research log, for a researcher investigating William Frazer. Columns include dates, place of research, purpose, call number, source, document number and results. The bottom of the log includes a research question and suggestion, which help to shape the research.
    Example of a handwritten research log – I tend to prefer wider spacing than this so it’s easier on the eye. This is only one example of a log: you can develop a version which works for you. Source: FamilySearch Wiki.

    If you’re new to research logs, then Cyndi’s List has some great resources for you to explore on the subject. Natalie Pithers of Genealogy Stories has also written a great overview of logs and why you need them on her blog.

    5. Beware anchoring from oral histories

    Oral histories (evidence drawn from conversations rather than written documents) are typically a first port of call in our genealogical journey. How many of us started our family history journey by talking to our older relatives? Conversations can be a rich source of information about our forebears, but are unlikely to be 100% accurate. The nature of oral histories as a frequent starting point makes them a particular source of anchoring bias.

    For instance, I’ve encountered cases where someone was adamant they knew their mother-in-law’s maiden name, but in fact got it completely wrong; cases where a family story about a particular religious affiliation turned out to have no basis in fact; and stories of connections to famous people which weren’t true at all. So as with all our research, BE CRITICAL!

    Oral histories can be an amazing source of information, but also a major source of anchoring bias. Photo credits: Whisper by ElizaC3, CC BY 2.0; Anchor by Phong6698, CC BY-NC-ND 2.0.

    Once you’ve recorded the oral history you’ve taken from a relative, try asking yourself: “What does this oral history suggest I should look at?” followed by, “If some of these ideas are inaccurate, what are my alternatives?” This way, you’ll have a Plan B in mind from the start and might be less likely to be derailed by misinformation.

    Be especially careful not to assume all information you’re told is accurate. Use it as a guideline to suggest research directions, but don’t assume that it provides the complete picture.

    6. Celebrate context

    Ask “what are my blind spots?” and approach genealogy as a perpetual learner. Delve into some background reading about the era and place that you’re researching; understand the provenance of the record sets you work with. Appreciating context in this way will really improve your judgments when navigating the records.

    7. Practise, practise, practise

    The final step in battling our biases is to continue on our research journey with these bias-battling tools in mind. There is no quick, one-stop fix for cognitive bias, but over time you can train yourself to spot when it’s happening. 

    It’s a reassuring thought that, whatever level of experience we bring to our genealogy research today, we’re all learning as we go. The more secure you become in your genealogical technique, the better equipped you’ll be to address bias in your own practice – as well as helping friends and acquaintances with theirs!

    Try some of the suggestions above and you’ll find that bias-battling gets easier with time. Critiquing our own work can help our research to flourish, gifting us family history stories that we’ll enjoy sharing with our families and descendants for many years to come. 

    What’s your advice?

    Do you have any favourite bias-battling approaches or advice you’d like to offer to your fellow genealogists? Or perhaps you’d like to share your own experience of being led astray by bias? Post your tips and stories in the comments below and perhaps you’ll help others avoid the pitfalls of the genealogical elephant in the room…!

    Further Reading

    If you enjoyed reading about cognitive bias and what it might mean for your genealogy research, then I’d highly recommend the following articles and resources for further reading.

    BOOK: Thinking, Fast and Slow by Daniel Kahneman. This bestselling book comes from one of the original academics to identify and characterise cognitive bias – it’s a great introduction to the major concepts.

    BOOK: History in Practice by Ludmilla Jordanova. Chapter 5, on “The Status of Historical Knowledge”, provides a great overview of the reasoning processes that underpin historical research.

    ARTICLE: Confirmation Bias and the Power of Disconfirming Evidence by the Farnam Street blog. Their Mental Models page, which provides more information about cognitive biases and how to challenge them, is also worth a look.

    ARTICLE: The Cognitive Biases Tricking Your Brain. Ben Yagoda’s article for The Atlantic looks at some of the history of our awareness of cognitive bias and studies on how we might unseat it.

  2. What is Proof?

    Leave a Comment
    This excellent article was written by Phil Isherwood, and he has kindly allowed us to reproduce it here. Phil is a Pharos Graduate who describes himself as an amateur genealogist. His daily working life includes developing methodologies for business teams, a skill transferable to genealogy research. You can find Phil’s blog here: Seeing the Wood for the Trees and follow him on Twitter: @isherwood_phil 
    The end goal of all genealogical investigations is to establish proof, by which we mean a convincing, credible case for a specific ancestral identity, relationship, or life event. By thorough research in sources, we must find sufficient detailed, matching evidence to uncover and reconstruct relationships and events relating to our research target, and only our research target.
    But how do we decide when an accumulation of evidence reaches a threshold that we call proof? How do we define it? How do we decide when it has been met? How do we assess what others have proposed as proof? Is there just one valid definition of proof or are there many? These are questions that are central to the pursuit of genealogy.
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    Why is a definition of proof important for genealogy?

    Once, in those not so distant pre-Internet days, many genealogists laboured away in near isolation, uncovering family histories from public, private or academic sources then sharing the outcomes within their families. Today, genealogy is a collective activity, nourished by easy online access to many sources, supported by vibrant societies and online communities. We share ideas and knowledge, we communicate with and educate one another, and, crucially, we often share our findings. One need only take a brief look at the thousands of published trees on Ancestry and other large subscription sites to understand the ubiquity of sharing amongst family historians. Some, such as the LDS Church, have even gone as far as to suggest that an end goal of collective sharing could be a single, comprehensive family tree of everyone.
    But, as many who have looked at shared online trees in any detail, one quickly realises that the published conclusions of some family historians can stretch credulity. At best it can be said that there is a variable interpretation of what is required to establish a genealogical proof.
    A standard definition of what constitutes proof is clearly necessary for any serious genealogical endeavour, as without a consistent measure of proof we can’t achieve consistent outcomes. When collaborating with others, it is an essential.

    A History of Genealogical Proof

    In the UK there has never been an official definition of genealogical proof – something which I have found troubling. In the USA however, the Board for Certification of Genealogists (BCG) has been a leader in setting out formal definitions and, where needed, updating them over time.
    The BCG’s original definition of genealogical proof was based on the legal standard of proof as used in civil court cases. This principle is called the Preponderance of the Evidence, which amounts to “when I weigh all of the pros and all of the cons, I think that the pros outweigh the cons.”

    This Photo by Unknown Author is licensed under CC BY-NC-ND

    Sounds reasonable, right? But there’s a catch – the margin by which the pros outweigh the cons can be tiny, even infinitesimally small. Where there is a significantly larger quantity and quality of evidence on one side this can work, but in marginal cases this can lead to conclusions which we might view today with some suspicion.
    In the late 1990s, the BCG recognised the weaknesses of Preponderance of the Evidence and developed a new definition of called the Genealogical Proof Standard (GPS), which they published in 2000.

    The Genealogical Proof Standard

    The GPS is a five-step process which defines a well conducted genealogical investigation. It can be simply summarised as: search, cite, analyse, consolidate, and conclude. To have met the standard, the researcher must show that their whole investigation (not just the conclusions) meets all five of the elements.

    The Genealogical Proof Standard

    1. Reasonably Exhaustive Search
    Full text: “Reasonably exhaustive research – emphasizing original records providing participants’ information – for all evidence that might answer a genealogist’s question about an identity, relationship, event, or situation.”
    So, what is a “reasonably exhaustive search“? Simply put, it is a search that has examined all potentially relevant sources. It implies that we start our investigation by finding out what the potentially relevant sources will be, list them and then search them all in turn, consulting the original or an image of the original wherever possible. By doing so we minimise the risk of undiscovered evidence emerging later to overturn an initial, perhaps hasty, conclusion.
    2. Complete, Accurate Source Citations
    Full text: “Complete, accurate citations to the source or sources of each information item contributing – directly, indirectly, or negatively – to answers about that identity, relationship, event or situation.
    Thorough, accurate citing of sources helps us to remember where we found the information we rely on as evidence and enables others to validate that our search has indeed been “reasonably exhaustive”. Assuring others of the quality of our search is doubly important, it enables them to:

    • replicate our steps; and
    • contribute their own ideas about other relevant sources which could enhance our research.

    Citation is the single most effective tool for enabling effective collaboration between genealogists.
    3. Analyse and Correlate Sources, Information and Evidence
    Full text: “Tests – through processes of analysis and correlation – of all sources, information items, and evidence contributing to an answer to a genealogical question or problem.
    What this really means is that we need to make a sound interpretation of the evidence at our disposal. Kick the tyres, make sure that it stands up to scrutiny, be honest with ourselves about any gaps or deficiencies that may be there. This is the right point to consider whether the evidence we’ve collected forms a full and coherent picture. Is it the best available evidence? Is there any other potential evidence we could add to our search to strengthen our case? Is the evidence sufficient to support our conclusions? Will our conclusions reflect all the relevant evidence, good and bad, direct and indirect, positive and negative?
    4. Resolve Conflicting Evidence
    Full text: “Resolution of conflicts among evidence items pertaining to the proposed answer.
    It is a genealogical truism that any sufficiently exhaustive search will uncover at least some conflicting evidence. The corollary is also often true, that if you’ve failed to find any conflicting evidence then you may not have performed a sufficiently exhaustive search!
    Analysing and resolving conflicting evidence is an essential step. Are we able to understand what the conflicts in the evidence might mean? Can we account for them? Or does the conflicting nature of the evidence put our conclusion into doubt? If we’re unable to resolve conflicting evidence satisfactorily then we will not be able to formulate a credible conclusion.
    5. Soundly Reasoned, Coherently Written Conclusion
    Full text: “A soundly reasoned, coherently written conclusion based on the strongest available evidence.
    At first glance, this element of the GPS seems like a non-sequitur. Conclusions must be:

    • soundly reasoned – as no-one would accept a conclusion that relied on unsoundreasoning;
    • coherently written – as no-one would accept a conclusion that was written incoherently; and
    • based on the strongest available evidence – as no-one would accept a conclusion based on partial, weak, or inaccurate evidence.

    What this really means is that our conclusion must be based on a sound appreciation of what evidence was available, that we accurately interpreted and collated the evidence, and show how the evidence leads to the conclusion. It enables us to demonstrate that our conclusion is not only valid, but free from bias, preconception, or inadequate appreciation of the evidence.

    What the GPS is – and what it isn’t!

    There is little doubt that the Genealogical Proof Standard is a significant improvement upon Preponderance of the Evidence. It sets a far higher standard for proof to be achieved – no more dodgy marginal cases – and roots its requirements in the language of genealogy rather than a legal framework which has doubtful relevance to our pursuit. It is applicable to all genealogy work, not only in the USA but all around the world, whether professional or amateur. It sets clear expectations on how we should plan, execute, and document our work. It creates a common standard and supports common outcomes that can be scrutinised, replicated, or refuted. It is a vital tool that all genealogists who have mastered basic sources should try to understand and engage with. It is the missing link that may, in the future, allow genealogy to be the truly collective experience that it could and, many would argue, should be.

    This Photo by Unknown Author is licensed under CC BY

    But it has problems too:

    • it isn’t a single, cogent statement against which a documented conclusion can be measured;
    • there is no straightforward checklist one can use to say “if these items are covered in the documented findings then it passes”;
    • to assess whether a documented outcome has met the standard, the assessor must have access to details of how the investigation was conducted;
    • the five steps of the GPS each have detailed definitions which require some knowledge and skill to understand fully and utilise.

    One can easily use the GPS to assess formal genealogy reports, but it is simply not possible to use it to assess the principal form of communication in modern genealogy – the online tree.
    So, if it has shortcomings should we be making efforts to use it? Yes! Yes! Yes!
    The GPS is the only agreed standard for genealogical proof. It is the best available and we should all be using it so that:

    • we have confidence in our own conclusions;
    • we have confidence in the conclusions of our peers and collaborators; and
    • we can share our work in the confidence that it can be used as the basis for further investigation without the need to be re-verified from top to toe.

    It is a sad fact that in the laissez-faire world of shared online trees, one must re-assess all findings before using any element for one’s own purposes. Consciously using the GPS can reduce needless rework and, most importantly, make us all better genealogists.
    The GPS is not perfect – even investigations that follow it thoroughly and accurately can’t ensure perfect certainty in their outcome. (We are engaged in family history, not mathematics, so there can never be perfect certainty!)
    It is a little daunting, but it is the best framework and standard that the global genealogy community has. Once mastered, it can and will save you time, effort, energy, and tears!
    Have I persuaded you that the GPS is the best way forward for your genealogy? Please let me know in the comments below.

    Sources

    1. Board for Certification of Genealogists, Genealogy Standards (Board for Certification of Genealogists, 2nd Edition 2019).

    Personal note: This is one of the single most important, yet least readable, books in the history of genealogy!

    1. Anderson, Robert Charles. 2019. Tools for Testing Genealogical Proofs. Talk delivered at RootsTech London, 24 October.