Final Results: The Early-2022 AVM Bot Challenge

About 6 months ago, we announced our latest effort to test all the online valuation bots against the real world.

We tested the valuations for 14 local properties as presented on three popular websites: Zillow, Redfin and Realtor.com. (These estimates are often referred to as AVMs for "Automated Valuation Models.")

Could algorithms properly assess or predict Manhattan Beach home values?

Was any one system better, or worse, than the others?

If you'd like, you can refer back to our two prior posts on the topic ("The Early 2022 AVM Bot Challenge" and "How Are the AVM Bots Doing?").

We chose 14 listings that were active as of late January, most new to market. At the end of the day, 13 of the 14 listings sold, so we have plenty of data to go by. See our analysis of the AVMs and "report card" below.

-----------------------------------

NOTES:

Nerdy note: One of the listings actually canceled off the MLS instead of selling, but tax records show that it sold soon after. So we are counting that sale, since our purpose is to see how close the bots were to the sale prices. One other listing switched agents and began with a different list price before selling, but we compare its sold price here to its first list price and the initial bot estimates.

CYA note: Both Redfin and Zillow operate brokerages which could be viewed as competitors. It is generally not a good business practice to publicly critique competitors. We are not focused here on either brokerage's business practices, however, but rather the reliability of their publicly displayed property valuations, which seem like fair game for discussion.

Please Note: The properties subject to this test and listed below come from the whole Manhattan Beach market via the MLS, and were, or are, listed and brokered by a variety of agents and firms. The name of the listing broker and agent for each property, along with any contact information supplied through the MLS, is displayed with each property, next to the price. Blog author Dave Fratello of Edge Real Estate Agency is not the listing agent or buyer's agent unless so stated with the listing. Details on every property are provided below, along with comments.

-----------------------------------

Report Card: Rating the AVM Bots

It's Summer, and maybe no one wants to think about report cards. But frankly, some of these bots need to go back to school.

Oddly, it's Realtor.com's menagerie of multiple valuation bots that appears to need the most remedial training.

That grade of D is possibly too generous. As you'll see below, 60% of Realtor.com's estimated values were more than 10% off from reality, with a whopping 43% of the estimates off by 15% or more.

That's bad, but it gets worse. One out of four Realtor.com estimates were off by more than 40%. If you put the same kind of mathematical rigor into building a rocket, not only would you fail to get off the launch pad, the whole jalopy would topple over before it was ready. People could die!

This is all the more amazing because Realtor.com posts 3 separate valuations for each property, using different methodologies. You might think that having more chances to be right would increase the odds of being right, but no. For Realtor.com, it's like having 3 lottery tickets instead of one... nearly equal odds of being a loser.

(In our chart below, we are rating all 35 valuations given by Realtor.com on 13 sold properties. If we pick only the "most accurate" out of 3 valuations by Realtor.com for each of 13 properties, the results still are similar.)

Redfin's bots, by contrast, can enjoy the rest of Summer on a sandy beach somewhere. (Just don't get wet!) They posted a remarkable record of accuracy among these Manhattan Beach listings, compared to the others.

Redfin was quite a bit more likely than not to suggest a value within 5% of the eventual sale price. Take note of the very big red 46% here, indicating that a large number of Redfin's estimates were between 2-5% of the eventual sale price.

An automated value within 5% of the real value is good, no question.

We have noted before some of our skepticism towards various AVMs, which sometimes seem to "cheat off the listing agent's paper." That means these figures often seem to be close to the public MLS asking price. If the AVM factors in the list price as a data point, the system isn't really assessing inherent value or comps, but taking advantage of the work that was already done by the listing broker. (We don't actually know how these algorithms are built, we just have suspicions.)

An actual real estate professional person, using some combination of experience, street smarts and MLS data, ought to be able to come within 5% most of the time - much more than 61% of the time, it is to be hoped. But hey, we humans get the advantage of seeing houses in person, being able to use subjective factors and just generally being able to think critically. (In most cases.)

Now, you probably wouldn't hire as your broker a person who had a track record in which 30% of the time they totally blew it, missing the proper value by 10% or more. But it's decent for a bot.

Zillow pretty much invented online, automated valuations for the mass public, but they just aren't hitting their mark here in Manhattan Beach.

Our C+ grade here also feels a bit generous, and it's really showing how we grade on a curve.

Zillow had only 25% of its estimates within 5% of the sale price, with another 25% being within 5-10%.

Put simply, there was a 50-50 chance that the Zillow number for a Manhattan Beach property was within 10%. And then, a 50-50 chance that it wasn't within 10%. 

This performance isn't even close to the track record Zillow purports to have in the "metropolitan area" of "Los Angeles-Long Beach-Anaheim." They have crunched their own numbers, and Zillow says on this page that they're within 5% of the price 61% of the time, and within 10% of the price 88.5% of the time.

You say 88.5%, I say 50%, who's to say who's right?

The first time we publicly tested Zillow estimates for Manhattan Beach, back in 2013-14, their system also got within 10% of the sale price exactly half the time.

Plus ça change, plus c'est la même chose.

That's French for: If you want a property valuation done right, don't rely on a corporate bot to do it for you.

 

About the Properties in the Test

Here is a full list of the properties that we pulled out for the test in January.

List price, sale price and automated valuations are all provided, along with brief comments.

200 40th Street (4br/5ba, 2468 sqft.)

Start price: $5,499,999

Sold price: $5,300,000 (-$199K / -4%)

Listed by Matt Waxman of Palm Realty Boutique, Inc. 

Estimates

Zillow: $1,964,500

Redfin: $5,463,632

Realtor.com: $2,002,700 / $2,267,907

Comment:

Nothing like this sale has ever happened in El Porto. Huge number.

Zillow and Realtor.com seemed to think it was an old/original duplex.

Remember how we said the AVMs sometimes appear to cheat by factoring in the list price?

How else did Redfin get within 3% of this unprecedented sale?

1508 Magnolia Avenue (5br/2ba, 1896 sqft.)

Start price: $2,200,000

Sold price: $2,650,000* (+$450K / +20%)

(*MLS listing canceled)

Listed by David Caskey, eXp Realty of California Inc | 310-374-1800

Estimates

Zillow: $2,136,500

Redfin: $2,298,652

Realtor.com: $2,198,000 / $1,981,600 / $1,858,432

Comment:

Everyone missed the idea of this expanded cottage going over $2.6M. It was early 2022.

1516 The Strand (4br/4ba, 4953 sqft.)

Start price: $17,500,000

Sold price: $16,200,000 (-$1.300M / -7%)

Listed by Kurt Rappaport of Westside Estate Agency Inc.

Estimates

Zillow: $10,888,200

Redfin: $16,631,767

Realtor.com: $7,918,000 / $7,993,107

Comment:

Sometimes it takes 18 months or more to sell a Strand property, as happened here.

Redfin was within 3% again, while the others utterly blew it, with Zillow off by 33% and Realtor.com missing by more than 50%.

501 5th Street (4br/6ba, 4483 sqft.)

Start price: $4,488,000

Sold price: $4,475,000 (-$13K / -0.3%)

Listed by David Keller of Re/Max Estate Properties

Estimates

Zillow: $6,830,400

Redfin: $4,491,847

Realtor.com: $5,604,000 / $2,161,000 / $4,128,431

Comment:

We are still vastly entertained by the cacophony of prices suggested here, with only Redfin (again) close.

Imagine a room full of kindergarteners. Ask them "What's old?"

10!

145!

40!

That's what these numbers mostly look like.

341 4th Street (2br/1ba, 879 sqft.)

Start price: $3,899,000

Sold price: $4,554,000 (+$655K / +17%)

Listed by Bryn Stroyke of Bayside

Estimates

Zillow: $3,899,000

Redfin: $3,901,506

Realtor.com: $2,437,000 / $2,581,100 / $1,975,817

Comment:

Zoinks. Realtor.com doesn't know anything about walkstreets.

Zillow literally adopted the list price, but that was too low.

Redfin at least tried to hide things by adding a random $2,506 to the list.

The house went $655K over list.

1500 Walnut Avenue (3br/2ba, 1212 sqft.)

Start price: $2,399,000

Sold price: $2,600,000 (+$201K / +8%)

Listed by Jennifer Caras of Vista Sotheby’s International Realty

Estimates

Zillow: $2,518,200

Redfin: $2,535,894

Realtor.com: $2,385,000 / $2,447,100 / $2,218,439

Comment:

Everyone was pretty close, with Redfin suggesting the greatest overbid, $136K.

It went $201K over.

665 19th Street (5br/7ba, 3598 sqft.)

Start price: $4,899,000

Sold price: $4,674,000 (-$225K / -5%)

Listed by David Caskey, eXp Realty of California Inc | 310-374-1800

Estimates

Zillow: $5,143,600

Redfin: $4,893,785

Realtor.com: $5,156,000 / $4,935,600 / $4,913,273

Comment:

Absolutely none of the 5 estimates were under asking.

The home sold under asking.

42 Dover Place (3br/3ba, 2042 sqft.)

Start price: $1,950,000

Sold price: $2,011,600 (+$61K / +3%)

Listed by Vanessa Sokolsky of Harcourts Hunter Mason Realty

Estimates

Zillow: $1,891,100

Redfin: $1,927,612

Realtor.com: $1,943,000 / $1,960,600 / $1,990,989

Comment:

Two of Realtor.com's estimates were over asking.

The home sold over asking.

This was Realtor.com's best performance, the only one within 1%.

14 Monterey Court (2br/2ba, 1465sqft.)

Start price: $1,325,000

Sold price: $1,325,000

Listed by Brianna Franco of eXp Realty of California Inc

Estimates

Zillow: $1,347,200

Redfin: $1,358,435

Realtor.com: $1,441,000 / $1,383,200 / $1,453,369

Comment:

Shoulda stuck with list price, guys. The home sold for list price.

207 S Aviation Boulevard (2br/1ba, 906 sqft.)

Start price: $1,488,000*

Sold price: $1,300,000** (-$188K / -13%)

(* First listing ** Second listing/new agent.)

Sold listing Listed by Edward Kaminsky of eXp Realty of California, Inc

Estimates

Zillow: N/A

Redfin: $1,507,154

Realtor.com: $1,451,000 / $ 1,423,100 / $1,394,266

Comment:

It's hard to say anyone "won" this one. Zillow even sat it out.

In Realtor.com's favor, you can note that all their estimates were under the original asking price.

625 33rd Street (3br/3ba, 2080 sqft.) 

Start price: $2,499,000

Sold price: $2,680,000 (+$181K / +7%)

Listed by Casey Janssen of The L3

Estimates

Zillow: $2,551,300

Redfin: $2,554,755

Realtor.com: $2,489,000 / $2,517,400 / $2,108,835

Comment:

Three of 5 estimates were over asking, but none on the scale that the market delivered.

2501 Walnut Avenue (4br/4ba, 3766 sqft.)

Start price: $3,099,000

Sold price: $2,835,000 (-$264K / -9%)

Listed by Suzanne Dyer of Berkshire Hathaway HomeService

Estimates

Zillow: $3,102,000

Redfin: $3,023,275

Realtor.com: $4,088,000 / $3,219,600 / $3,480,422

Comment:

All the bots blew this one, unable to fully factor in the presence of busy Marine Ave. right next to the entry door and living room/dining room of this house.

Of course, the sellers and listing agent also estimated high... by 9% in this case.

2612 Alma Avenue  (3br/2ba, 1233 sqft.)

Start price: $2,999,990

Sold price: $2,600,000 (-$400K / -13%)

Listed by Don Ruane of Real Estate West, Inc.

Estimates

Zillow: $3,000,000

Redfin: $3,009,198

Realtor.com: $2,966,000 / $2,998,600 / $3,001,976

Comment:

All the AVM bots believed too much in the list price, hewing close to it in their estimates.

But the listing started 13% too high, judging by the sale price.

Not Yet Sold

3216 The Strand (5br/8ba, 7454 sqft.)

Start price: $30,000,000

Listed by Christa Lyons, eXp Realty of California, Inc.

Estimates

Zillow: $16,480,600

Redfin: $27,191,401

Realtor.com: $11,400,900

Comment:

This was the most expensive Strand listing ever when it emerged early in 2022. It might take some time. Now there's an even pricier listing at $36M on The Strand.

The bots have never seen anything like this, with two far, far away from asking.