Wrapping Up Our Zillow Test

Late last Summer, MBC began a test of Zillow's accuracy by seeing how their Zestimates of newly listed properties worked out in the real world. (See "Testing Zillow's Estimates.")

We began with a snapshot of the 12 (then-) newest listings in mid-August 2013, then tracked progress on those over time. We're now 6 months past the start of the test, and ready to call it a wrap.

Before we ask how, exactly, Zillow performed, let's recall how well Zillow says they expect to perform.

As you can see on this page, Zillow's engineers publish their own statistics on the accuracy of Zestimates. In the LA Metro area, for Q3 2013, the Zestimate at the time of a public listing's launch was within 5% of the eventual closed sale price 32% of the time.

Rephrase that: One-third of the time the Zestimate is reasonably accurate.

So how'd they do in MB? Here's the chart.

Out of 10 solds, Zillow was within 5% on 1027 Boundary, 1300 12th #D and 1336 2nd St.

That's a 30% accuracy rate, exactly as advertised. And each of these was actually within 2% – when they're right, they're right. For the condo at 12th St. (pictured), Zillow's screaming line-drive home run, they were within $492.00.

In addition, Zillow was within 10% on 509 Dianthus and 1757 Voorhees. That brings Zillow to 50% within 10%, close to the LA County average of 58%.

Zillow failed spectacularly on three of these listings: 1202 Tennyson, 1509 Magnolia and 594 33rd. That was due to poor information, but we fully expected that, and wrote those off in our first post. (On Tennyson, Zillow had mega lot size for the whole condo complex, not just the unit; on Magnolia, the robots didn't know the home was completely remodeled and, on 33rd, they didn't know it was new construction.)

So one lesson there would be, if it is clear that Zillow is missing an important detail, you should ignore the Zestimate completely. (If we had actually dropped these from our data set at the beginning, due to these obvious input errors, Zillow's success rates would be far higher.)

Finally, Zillow wasn't very close on 25 Sausalito Circle (super remodeled) or 112 18th St. (pictured), though both pricing errors were "understandable," as we explained in our first post.

One of the three big "understandable" gaps in our initial post was between the sellers and Zillow on 1027 Boundary, where the sellers began at $1.980M but Zillow was thinking more like $1.8M. Turns out both were high, and the $1.770M closed price was 11% under the start and just 2% under the Zestimate.

A little pat on the virtual back is in order for the robots on that one, which we had said was "very hard to value, given the unusual style, location and changes in the market since last year [when the home had last traded]."

So, can you trust the Zestimate on a home? It depends. If they didn't obviously botch the basic details, there's a good chance the algorithms are reasonably close – and some chance they're dead on. That's a starting point, anyway, before you look to other sources.