How Are the AVM Bots Doing?

By Dave Fratello | March 16th, 2022

About 6 weeks ago, we announced our latest and most ambitious effort yet to test all the online valuation bots against the real world of Manhattan Beach real estate.

In that first post ("The Early 2022 AVM Bot Challenge"), we took a snapshot of 14 active listings and shared the valuations for each property on three popular websites, Zillow, Redfin and Realtor.com.

Now it's time to see how well the bots have done so far.

With 6 of the 14 properties having closed, we can give out some initial grades.

Spoiler alert: Not one of the Automated Valuation Models (AVMs) did too well.

Missed The Rising Market

For these first two closed sales, many automated valuations hovered close to the asking price, but the properties sold well over asking.

341 4th Street Manhattan Beach CA341 4th Street (2br/1ba, 879 sqft.) is a South End walkstreet lot sale.

Values for walkstreet plots like this have hovered in the mid-3s for years, and recently started hitting the higher 3s.

This one, however, blew away all records.

Start price: $3,899,000

Sold price: $4,554,000 (+$655K / +17%)

Yowza! Did any of the bots see that coming?

No one did, to be fair. But no, no bots did.

Estimates

Zillow: $3,899,000

Redfin: $3,901,506

Realtor.com: $2,437,000 / $2,581,100 / $1,975,817

Comment:

Realtor.com looked ridiculous from the start, even though they took 3 guesses. Their valuations obviously did not take stock of the walkstreet location and how special that has proved to be for values in recent years. All of the Realtor.com estimates were more than 50% too low.

Zillow and Redfin basically cheated off the listing agent's paper, running with the list price, more or less. So they were off by 17%.

1500 Walnut Avenue Manhattan Beach CA1500 Walnut Avenue (3br/2ba, 1212 sqft.) is a pretty, pretty remodeled cottage in the Tree Section

Start price: $2,399,000

Sold price: $2,600,000 (+$201K / +8%)

Estimates

Zillow: $2,518,200

Redfin: $2,535,894

Realtor.com: $2,385,000 / $2,447,100 / $2,218,439

Comment:

This home didn't have the unusual feature of being on a treasured walkstreet, and came with a reasonable set of comps to go by.

Zillow and Redfin both estimated that the house would sell substantially over asking. Redfin was boldest, expecting a 5.6% markup, but it was 8% in the end. We'll credit these two bots for getting the direction right, if not quite the full scale of the overbids that were going to come.

Realtor.com lagged again, with one out of three guesses being a little over asking, but still $153K short. Two of their guesses were 8% and 15% low, respectively.

Pretty Close

Here are 3 closed sales where at least one of the AVM bots came close to being correct.

501 5th Street Manhattan Beach CA501 5th Street (4br/6ba, 4483 sqft.) is a large 1990 house that's on a corner among the South End walkstreets. There's a lot to work with here, but that also means considerable expense to update.

Start price: $4,488,000

Sold price: $4,475,000 (-$13K / -0.3%)

Estimates

Zillow: $6,830,400

Redfin: $4,491,847

Realtor.com: $5,604,000 / $2,161,000 / $4,128,431

Comment:

We're going to credit Redfin with a near-hit here.

Granted, they just tweaked the list price, suggesting $3K over asking, when in fact the home closed $13K under asking. Still, that was the right bet.

Zillow got so far out over their skis, they were tumbling down the mountain and into the village. No, the home did not close 52% over asking at $6.8M. Epic fail.

Realtor.com threw three darts wildly, and none wound up close. Their closest was 8% low, another was 25% too high, and ... $2.1M? Not worth another minute.

Y'know, at this point, Realtor.com looks like their bots just aren't very good at this, even though they've got more guesses than anyone else. Did management skim away the engineering budget?

42 Dover Place Manhattan Beach CA42 Dover Place (3br/3ba, 2042 sqft.)

Start price: $1,950,000

Sold price: $2,011,600 (+$61K / +3%)

Estimates

Zillow: $1,891,100

Redfin: $1,927,612

Realtor.com: $1,943,000 / $1,960,600 / $1,990,989

Comment:

This Village Plan 7 townhome had generated pretty close agreement among the bots at the outset.

Of course, most were hewing close to the list price.

Only one guesser bot predicted that this property would sell over asking: Realtor.com's, twice. They thought it would be over by $10K or maybe $40K. It was actually $61K, but this is by far Realtor.com's best performance, so we'll give the bot a little pat on the... microchip. They were within 1% on one of three guesses. Bravo, little dude.

Zillow and Redfin both assumed the seller was asking too much, and would get taken down a notch. Nope. Zillow was 6% low and Redfin was 4% low.

Estimated Too High

It was uncommon among the 4 closed sales above to see any of the AVMs guess too high, with exceptions where the bots just wildly misfired.

Here are two cases where the bots were kinda close to asking price, but buyers paid less.

2501 Walnut Avenue Manhattan Beach CA2501 Walnut Avenue (4br/4ba, 3766 sqft.)

Start price: $3,099,000

Sold price: $2,835,000 (-$264K / -9%)

Estimates

Zillow: $3,102,000

Redfin: $3,023,275

Realtor.com: $4,088,000 / $3,219,600 / $3,480,422

Comment:

This spacious early-90s Tree Section house came out in January and had the market to itself, but struggled regardless.

That was mainly because of the location along busy Marine Ave. You can watch traffic pass from the main living spaces downstairs.

Realtor.com's bots were ultra-bullish, dead-certain that this would sell over asking. It was only a question whether that would be $120K over asking, $381K or (yikes) $989K.

Zillow predicted a $3K markup over the starting list price, and Redfin anticipated a discount of $76K from start. (Alas, both were unlikely by the time we wrote our post and captured these AVMs, as the price had already been cut $104K to $2.995M.)

Everyone was high, everyone was wrong when the home sold for $2.835M.

Realtor.com was high by 13%-44%, Redfin was high by 7% and Zillow was high by 9%.

The reason they all guessed high was failure to account for location, an issue we anticipated in our first post:

It's only, like, the most cliché (and important) thing about real estate. "Location, location, location!"

If you can't turn that into math, why even try to automate home valuations?

14 Monterey Court Manhattan Beach CA14 Monterey Court (2br/2ba, 1465 sqft.) is a simple Plan 1 court home in Manhattan Village.

The bots were nearly at their best with the other Manhattan Village listing noted above.

How did they do here?

Start price: $1,325,000

Sold price: $1,325,000

Estimates

Zillow: $1,347,200

Redfin: $1,358,435

Realtor.com: $1,441,000 / $1,383,200 / $1,453,369

Comment:

Wrong, wrong, wrong. Every one of the bots shot high. All of them were wrong to do so.

Out of 5 guesses, couldn't someone have just said "list price!" No one did. But the home sold for list price.

Zillow and Redfin were about 1-2% too high. Realtor.com's guesses ran 4%-10% too high.

They tried to make it simple, and you guys just had to make it hard!

Initial Grades

We're not quite halfway through the randomly chosen, late-January listings that we used for this test. A picture is emerging.

Realtor.com: D+

There was virtually nothing accurate about Realtor.com estimates. It's maddening because they give themselves three guesses on most listings, yet still consistently fail to hit the mark.

We've seen Realtor.com estimates here that were 50% too low and as much as 44% too high, with lots in between. Only once were they within 1% of the sold price, the one time when all three estimates were over-asking while the other AVMs guessed low.

A D is a failing grade, just like an F, but we want to provide a little encouragement. Based on one performance only (42 Dover Place), we'll try to offer a little encouragement, so that they keep coming to class and maybe turn it around.

Zillow: C+

We're feeling charitable here.

Zillow did have one honker in here, but otherwise was within a few-to-several-percent of most sale prices.

Now, mainly that's because Zillow often tracked the list price, with their estimate near asking or just a little over or below. We can't rule out that their algorithm gives a lot of weight to the list price, also known as the professional opinion of a real person who actually works in the field.

So while this may not validate any innovative algorithms that Zillow has been refining for years and years, it at least seems to suggest that consumers can find the Zestimate to be relatively reliable.

Then again, you could skip the middleman and just place a degree of faith in the list price.

Redfin: B-

We're taking much the same approach in grading Redfin as we did Zillow.

Redfin lacked a honker and was the closest on two properties (501 5th and 1500 Walnut).

We recognize that they, too, must be factoring in the list price in making their estimates, but as a result, they're reasonably close-ish.

If as a consumer you want results, meaning estimates that are somewhat reliable-ish, Redfin is the most passable from this set.

Now, if yours is the kind of home where you don't even count grades that aren't A's, you're not going to be able to put much faith in the bots right now.

-----------------------------------

CYA note: Both Redfin and Zillow operate brokerages which could be viewed as competitors. It is generally not a good business practice to publicly critique competitors. We are not really focused here on either brokerage's business practices, however, but rather the reliability of their publicly displayed property valuations, which seem like fair game for discussion.

Please see our blog disclaimer.

Latest Listings Among
Manhattan Beach Homes For Sale