narrative vs. fairytale
making decisions with data
storytelling is a fundamental component of humanity
language evolved to share social information
stories are how we make sense of the world
stories are how we decide what to believe
the stories we tell are how we make decisions
reassert humanity in decision-making
humanity in engineering
how does this relate to tech?
if you work in content, product or engineering,
you exist in a series of stories
what's wrong with fairytales?
they're not real
definitions
data
discrete, objective facts (quantitative or qualitative)
story
connecting disparate data
definitions
data
what we know
story
what we think
penguin yellow balloon
types of stories
narrative
fairytale
narrative
- explain data with logical connections
- state inferences and assumptions
- same dataset can have multiple narratives
fairytale
- connects data that does not relate to one another
- makes suppositions not supported by the data
- invents data out of whole cloth, or uses bad data
• what are common signs a story is a fairytale?
• how can we correct back to narratives?
preshow
not using data science
- statistical significance
- sampling bias
- average vs. median
no. 1
inventing or inferring explanations
- why questions about user behavior
- basic metrics used to describe complex interactions
data cannot tell you why
how to avoid this
- be clear that the explanation is an educated guess
- treat that guess as a hypothesis
- test that hypothesis (using other data or experiments)
- be willing for that hypothesis to be wrong
no. 2
load-bearing single data points
- assumed single variable
- common issue in engineering productivity metrics
- product analytics
campbell's law
the more emphasis you put on a data point for decision-making, the more likely it will wind up being gamed
goodhart's law
when a measure becomes a target, it ceases to be a good measure
no. 4
using data because it's available
- no. 1 biggest problem in corporate environments
- it's hard to say no when asked a direct question!
how to avoid this
- be up-front about the limitations of the data
- be willing to say "i don't know"
- collect the data needed to answer the question
- collect your data with intent
no. 5
ignoring context
- not factoring in common-sense guardrails
- always assuming historical data is relevant
how to avoid this
- be aware of the context of the data
- be aware of the context of the question
- be aware of the context of the decision
- be aware of who is asking
no. 6
discounting other possible explanations
- narrative is interpretation
- it's easy to get attached to a narrative
- multiple narratives can be true
- you almost never have all the facts
how to avoid this
- acknowledge other possible narratives
- call out any assumptions or extrapolations
- talk about the weaknesses or doubt of any data
data is not a substitute for
understanding people
(users or employees)
solutions
- don't rely on single data points
- consider the context
- collect data with intent
- don't be afraid of "i don't know"
- be explicit about assumptions and caveats
- data-informed, not -driven or -justified
and they lived happily ever after
thank you
questions?
hello@kait.dev
written version
https://kait.dev/fairytale