Testing Zillow Again

By Dave Fratello | January 24th, 2017

It's been almost 3 years since we ran a very public test here at MB Confidential of the accuracy of Zillow's pricing estimates ("Zestimates").

Now it's time to try it again.

The test is simple. We're taking the first 15 listings in the Manhattan Beach real estate market of 2017. We'll capture both their asking prices and Zillow's automated valuations for each one.

Shake, stir, wait a few months, and we'll revisit the listings to see how close Zillow was on each one that actually sells. (You can read our final post on the first Zillow test here: "Wrapping Up Our Zillow Test." [Feb. 2014])

In the time since our last public test, Zillow's own engineers claim to have made their system much, much more accurate.

In 2013, they were claiming that for the LA metro area, Zestimates were within 5% about one-third of the time (32%). Zillow said their Zestimates were within 10% of the sales price 58% of the time.

Now in 2017, Zillow makes bolder claims: Their Zestimates are within 5% almost 60% of the time (58.8%), and within 10% fully 80.4% of the time.

Notice that this means Zillow claims to have nearly doubled its accuracy in the "sweet spot," within 5% of the correct price. See LA at the bottom of this chart:


Also worth noting: Their "median error" rate for the area is 3.9%, which is to say that half the Zestimates for homes are within 3.9%, while half are off by more.

These claims are bold, and might reflect tremendous success in machine learning.

Zillow's engineers have been trying to give their Zestimate engines the tools to figure out real estate markets by themselves. They basically programmed their system to try one method, then another, and another, and another to come up with property value estimates... and keep score on which method is most accurate for a given area. This way, without ever visiting a local real estate market in person (ha ha!), Zillow's computers would gradually learn the most accurate methods of valuation to rely upon.

All of this sounds great, and then you test it. So here we go.

Here are the first 15 listings to come to market in Manhattan Beach in 2017, along with their current Zestimates:

1842 10th Street 5/6 4707 7504 $2,500,000   $2,955,477 01/19/17
412 Highland Avenue 3/4 2100 3334 $2,650,000   $2,466,703 01/18/17
409 3rd Street 3/1 1186 2705 $2,500,000   $2,702,306 01/17/17
716 12th Street 6/7 4475 3007 $4,690,000   $4,709,530 01/15/17
831 Duncan Place 4/3 1578 5318 $2,950,000   $3,032,117 01/13/17
1315 9th Street 6/8 5585 7396 $3,700,000   $3,305,559 01/12/17
2600 Grandview Avenue 6/8 4410 2700 $6,975,000   $2,594,739 01/12/17
599 35th Street 3/3 3078 5401 $2,495,000   $2,555,841 01/11/17
1 Tiburon Court 3/3 1972 2285 $1,490,000   $1,482,094 01/10/17
1240 5th Street 6/7 10009 0 $7,500,000   $7,681,486 01/09/17
2615 Laurel Avenue 4/4 3101 4329 $3,199,000   $2,872,893 01/09/17
1657 Nelson Avenue 5/6 4354 7500 $3,350,000   $3,322,311 01/05/17
1600 1st Street 2/1 756 4273 $1,199,000   $1,211,758 01/05/17
1535 11th Street 3/2 1745 6506 $1,599,000   $1,580,602 01/04/17
441 1st Street 5/5 4676 2799 $4,899,999   $5,209,932 01/02/17

2600 Grandview Avenue Manhattan Beach CAThere's only one obvious, egregious error here: 2600 Grandview is a newly constructed house, but Zillow doesn't seem to know that yet.

The active listing for the property displays fine on Zillow, but the robots think it's a land deal only, so they estimate around $2.6M instead of the asking price, near $7M for the finished house. Too bad about that mistake, because it would be valuable to get some input on the value on that unique property. 

If there's one big impression you get from comparing the Zestimates to the list prices, is that Zillow is remarkably close on most of them to the active MLS listings' asking prices.

In theory, Zillow is not depending on MLS asking price as a data input, but who knows? For about 2 years now, Zillow has been connected directly to our local MLS, pulling updated information directly from the same pipe that realtors tap, so maybe they are factoring in the asking prices somehow.

If Zillow is using MLS data that way, then we've got mixed feelings.

On the one hand, we imagine a world of pure science where Zillow's computers somehow figure out a valuation methodology that works as well as any human-directed process. What a brave new world!

On the other hand, if the use of MLS data is making Zillow smarter, and you value accuracy more than anything else (we value it a lot!), then maybe you'll think it's fine if Zillow is kinda cheating off everyone's papers.

On 8 of the 14 remaining listings, Zillow is estimating a sale price higher than the asking price. That's more than half! Looks like someone set the computers to "sunny."

The sellers of 441 1st, for instance, might ask their agent why they can't get the $300K more that Zillow says they can get. Then again, they know the answer, because they were on the market for months last year and didn't sell.

The newest listing here, 1842 10th, is Zestimated to be worth $2.955M, while the list price is $2.500M. We can't really see Zillow's reasoning here. They're higher than the MLS list price by 18% and really just high for the location, style, etc. Betting against that Zestimate panning out.

The South End Sand Section lot sale at 409 3rd came out at $2.500M, but that might well be a low price meant to draw multiple offers. We will see if that goes up from asking.

Zillow is a bit pessimistic about the new construction at 1315 9th, which is asking $3.700M, but which Zillow pegs at $3.305M, about 11% lower. One to watch, for sure, as this test unfurls.

716 12th Street Manhattan Beach CANewly in escrow is 716 12th St., listed at $4.690M in its second recent listing for resale. Zillow was thinking it was worth an extra $19K. The home was purchased new in April 2016 for $4.500M before the buyer evidently had a change of plans and put it back out for sale.

As the weeks and months go by, and various of these homes find buyers and close up, we'll check in on how Zillow has been doing.

Remember, they expect to get 60% of these "right" (9 of 15) by being within 5% of the eventual sale prices.

With their possible MLS data cheat, Zillow might yet pull that off.



Note: While we're tracking Zillow here, have you tested your own knowledge with our "Bet the House" contest yet? See this week's post and property here. Beat the crowd and win a nice prize!

Please see our blog disclaimer.

Latest Listings Among
Manhattan Beach Homes For Sale