We put ResiQuant's Engineering AI head to head with HazardHub, an incumbent third-party property data provider that many underwriting teams (and other AI workflow tools) rely on to enrich submission data. The comparison spans multiple building characteristics, but in this first release, we compare the accuracy of this traditional data provider vs ResiQuant’s Engineering AI on estimating the number of stories of a building.
The number of stories is one of the few things a catastrophe model needs to know about every property it assesses. Models use it to judge how a structure responds to earthquake shaking and how much of it is exposed to wind. It also feeds the floor area and replacement cost estimates that control TIV (Total Insured Value) and ITV (Insurance to Value) analyses.
How we tested it
We took a set of more than 1,000 buildings from real commercial submissions. For each one, we compared Engineering AI's story count with the count in HazardHub's property record.
Every answer was checked against imagery. A human reviewer counted the above-grade stories on each building from street-level and aerial views, without seeing either system's answer, and that count became the ground truth. Where sections of a building differed in height, the tallest section set the count.
We scored buildings of one to three stories on the exact count, and taller buildings with a one-story tolerance. Low-rise buildings are stiff, and a one-story difference changes how they respond to lateral load. Buildings of four or more stories are more flexible, so a one-story difference matters less to a catastrophe model. The tolerance also accounts for judgment calls, such as whether a parking level counts as a story.
The results
Figure 1. Share of buildings of two or more stories where each source had the right story count, against a ground truth counted from street-level and aerial imagery. Buildings of four or more stories count as correct within one story.
ResiQuant had the right count on 90% of buildings of two or more stories, against 57% for HazardHub (Figure 1). Where the two systems disagreed on a building, ResiQuant was right 85% of the time and HazardHub 22%, nearly four to one. The two add to more than 100% because on taller buildings, two answers one story apart can both fall within the tolerance.
Table 1. Share of buildings with the right story count. Buildings of four or more stories count as correct within one story, except in the exact-count row.
Across all buildings, ResiQuant leads by 19 points, 93% to 74% (Table 1). The lead depends on building height (Figure 2). On single-story buildings, the two sources score about the same. The largest gap is on two- and three-story buildings, 88% to 53%. On buildings of four or more stories, HazardHub missed 25% of the time, against 4% for ResiQuant.
Figure 2. Share of buildings with the right story count, by building height. Buildings of one to three stories are scored on the exact count. Buildings of four or more stories count as correct within one story.
Which direction a miss runs
The direction of an error, whether a source records too many stories or too few, matters when you evaluate a data source, because errors that run consistently one way bias the catastrophe model’s results.
In 92% of the cases where HazardHub missed, its enrichment of the number of stories underestimated the real number (Figure 3). In contrast, ResiQuant’s misses split almost evenly between too few and too many. HazardHub’s misses are also larger. It is off by three or more stories nearly five times as often as ResiQuant. In its two largest misses, it recorded both a fourteen-story building and an eleven-story building as one story each.
Figure 3. Misses by direction, as a share of all buildings in the test set. Bars left of the center line are buildings counted with too few stories, and bars to the right are buildings counted with too many. The rows below show how often each source was off by three or more stories, and HazardHub’s two largest misses.
What it means for a book
In a book shaped like this sample, HazardHub would be wrong on about 260 of every 1,000 buildings, and about 240 of those would enter the model shorter than they actually are. ResiQuant would miss about 70, most of them two- and three-story buildings.
The mix of building heights in a specific book changes those numbers. HazardHub and ResiQuant largely agree on single-story buildings, so a book weighted toward small buildings would see a smaller data accuracy gap. A book weighted toward taller buildings would see a larger gap.
A story count looks like the simplest field on a submission. But, like any building data, it is only as good as the source behind it.
