What is Reliability and Validity?
Reliability refers to the consistency of a measure. A trait assessment is considered reliable if it produces similar results under consistent conditions.
There are a few different types of reliability but the one we really focused on is internal consistency. Internal consistency tells us whether all the questions designed to measure a certain personality trait are indeed measuring the same underlying construct.
Think of it like this – if you have a bunch of thermometers in the same room, they should all give you roughly the same temperature reading. If they don’t, you wouldn’t consider those thermometers very reliable. Similarly, all the questions meant to measure a trait like extroversion should produce consistent scores. If they don’t, then the assessment may not be reliably measuring extroversion.
Our assessments have an average internal consistency score of 0.81 (Cronbach’s alpha). The scientific community views 0.6-0.7 as the minimum acceptable value, so our score of 0.81 demonstrates a very high level of internal consistency and thus reliability.
While reliability focuses on consistency, validity is about accuracy. Validity tells you how well a test measures what it claims to measure.
One way we assess validity is through convergent and divergent validity.
- Convergent validity checks that measures of constructs that should be theoretically related are indeed related. For example, measures of openness to experience and creativity should be correlated.
- Discriminant validity, on the other hand, checks that measures of constructs that shouldn’t be related are indeed not related. For example, extroversion and conscientiousness are distinct traits and their measures shouldn’t be highly correlated.
Evidence was gathered by evaluating correlational patterns using Structural Equation Modeling (SEM). Because these analyses are large, complex, and generally difficult to understand without training, we have opted to keep the data from the results in our SENSE technical report. However, the correlations with other traits provided strong support for convergence and divergence.
Another way we’ve validated our assessments is through confirmatory factor analysis (CFA). CFA is a statistical technique that allows us to test whether our data fits our theoretical model of which questions measure which traits.
Imagine you have a model car kit. The instructions tell you which parts fit together to make the body, the wheels, etc. CFA is like checking whether the parts actually fit together the way the instructions say they should. Do the extroversion questions actually fit together to measure extroversion.
To evaluate model fit (e.g., item relationships to the trait) we used multiple fit indices recommended by Hu & Bentler (1999), including 𝜲2, df, p, CFI, AIC, BIC, RMSEA, & SRMR. A review of the fit indices demonstrated good fit.
Finally we have collected criterion based validity through case studies with real entrepreneurs. Criterion-related validity tells us how well an assessment predicts outcomes in the real world. It s about ensuring that the traits we measure actually connect to the things that matter for your success.
For example, if our assessment identifies you as highly innovative and risk-tolerant, we want to see that those traits predict real entrepreneurial outcomes like launching successful products or securing venture capital funding.
By rigorously validating our assessments against these kinds of concrete criteria, we can confidently say that our insights don’t just describe you, but provide actionable information to help you succeed as an entrepreneur.