House Price Indices Compared

[FreeTrader]

We've had a number of threads over the years on the subject of house price index differences, but a couple of recent papers have clarified the position to a greater extent than in the past (IMHO).

I linked to the first of these (from the Insitute for Fiscal Studies) a few weeks ago:

Measuring house prices - a comparison of different indices

Today a briefing note from Acadata (compilers of the LSL/Acadata HPI - formerly Acadametrics) has added to the discussion, referencing the IFS paper above.

Which House Price Index?

The Acadata note understandably attempts to justify why their own index is best, but nevertheless this paper does have useful information and makes some good points.

The ONS has been working for some time on producing a definitive house price index for the UK, but thus far they haven't published an update on their progress.

[silver surfer]

Good post.

[FreeTrader]

In the OP I mentioned that the ONS is has been working for some time on developing a definitive house price index for the UK.

From today's HPI report we have the following update on progress:


In November 2013 it was announced that the four producers of official house price statistics
(Land Registry; Office for National Statistics; Registers of Scotland and Land & Property
Services, Northern Ireland) were continuing to investigate options for the production of
a single definitive UK House Price Index (HPI). This work was triggered in response to a
recommendation from the 2010 National Statistician’s Review of House Price Indices.

During 2014, work has focused on reviewing the available data sources and developing a
methodology that would allow a collaborative production of a new, definitive HPI. This work had
a clear goal to identify options for a HPI that would produce coherent house price data at the
UK, national, regional and sub-regional level, and covering all UK private residential housing
market transactions.

A draft methodology has been identified and each of the above organisations has agreed that
the work should progress to a consultation phase, where users of house price statistics will be
invited to comment on the intention to develop a new HPI and the proposed methodology to be
used.

The consultation will allow the proposed methodology to be refined where necessary and to
firm up the development costs for the production and implementation of a new HPI. These costs
will then be considered by each department against ongoing priorities and resource constraints
before a final decision is taken to implement the new HPI. Depending on the outcome of this
decision, it is anticipated that a new HPI would first be published in early 2016.

The consultation will be published in September 2014 and will provide users with details of
the proposed methodology, data sources, plans for production and publication, and the effect
the new HPI will have on the publication of current HPIs (assuming the new HPI development
moves to publication).

http://www.ons.gov.uk/ons/dcp171778_373873.pdf

[Venger]

We've had a number of threads over the years on the subject of house price index differences, but a couple of recent papers have clarified the position to a greater extent than in the past (IMHO).

I linked to the first of these (from the Insitute for Fiscal Studies) a few weeks ago:

Measuring house prices - a comparison of different indices

Today a briefing note from Acadata (compilers of the LSL/Acadata HPI - formerly Acadametrics) has added to the discussion, referencing the IFS paper above.

Which House Price Index?

That's an interesting paper, and you're right - it does dig much deeper in to how each organisation works out their hpi.

A good find.

They're all a 'tissue of lies' to me though. (A story invented in order to deceive). Market has been overridden by 0.5%, QE Trillions, FLS and everything else, to protect VI, and prevent what should have happened with house prices. And that should have, is not another 30%-100% HPI on top of 2007 peaks as we have now.

Oh well... Nationwide next week.

22 August 2014 Week ahead:

Lastly, and in the UK, the most noteworthy statistics will come on Tuesday and Thursday, with the latest figures on the number of mortgage approvals and outstanding drawings from the Funding for Lending Scheme (FLS) set for release.

Building society Nationwide will also publish its monthly data on house prices.

http://www.coutts.com/news-and-insights/newsitem/index.asp?nid=21985469

[FreeTrader]

They're all a 'tissue of lies' to me though. (A story invented in order to deceive). Market has been overridden by 0.5%, QE Trillions, FLS and everything else, to protect VI, and prevent what should have happened with house prices. And that should have, is not another 30%-100% HPI on top of 2007 peaks as we have now.

I certainly wouldn't go that far, but they do all have weaknesses.

The key point to bear in mind about house price indices is laid out by the authors of the IFS paper:

"In the case of ‘durable goods’ such as housing, we may be interested not just in the average value of houses sold in a given period, but also in the average value of the total stock of houses (of which those available for sale in any given interval will typically be a small proportion). As a result, it is important to be clear about what exactly a house price index is trying to measure: is it the average ‘price’ of housing transacted (or the flow of housing sales) over a specific period of time or is it the average ‘price’ of the stock of housing at a given point in time (or indeed the average price of a house that is in some way representative of the housing stock)? These different measures will not have the same value if the houses transacted in any period are not ‘representative’ of the housing stock in general – for example, if turnover rates of different types of property differ (i.e. some types of houses are bought and sold more frequently than others)."

In general the conclusion is that the main UK house price indices are more representative of the average price of houses that are transacting than they are of the average price of the housing stock.

Although indices use mix-adjustment to counter transactions bias, they don't altogether overcome it. Take the ONS index for example, where the weights are adjusted annually based on changes in the transactions mix.

This is what has happened to the geographical weights over the past few years:

As you can see, London's weighting has been increasing because sales there have held up better than they have elsewhere in the country.

ONS also gives a weighting to the type of neighbourhood a property is in. For this it uses the ACORN system, which is a geo‐demographic classification of all the neighbourhoods of the UK, produced by CACI International. Here's how these weights have changed:

These figures show increasing weighting is being given to sales in affluent neighbourhoods, where prices may be considerably higher nominally.

So, increased weight to London, and increased weight to wealthy areas in London - little wonder then that the headline ONS average house price has risen so much.

[porca misèria]

These figures show increasing weighting is being given to sales in affluent neighbourhoods, where prices may be considerably higher nominally.

So, increased weight to London, and increased weight to wealthy areas in London - little wonder then that the headline ONS average house price has risen so much.

Cart. Horse. Or do I mean Chicken, Egg?

The news stories talk of more cash buyers, fewer and smaller mortgages, especially in London. A London cash buyer is rather likely to be affluent

Ironic, when cash prices are high but costs of buying with a 20 or 25-year mortgage are close to an all-time low

[FreeTrader]

Cart. Horse. Or do I mean Chicken, Egg?

The news stories talk of more cash buyers, fewer and smaller mortgages, especially in London. A London cash buyer is rather likely to be affluent

Ironic, when cash prices are high but costs of buying with a 20 or 25-year mortgage are close to an all-time low

As it happens, the ONS index does not include any cash sales - it's based on mortgaged house purchases only, and specifically owner-occupier mortgages (i.e. BTL is excluded).

Clearly this introduces more potential bias.

Notably the ONS says that its aim with the new, definitive index it has been working on is to cover "all UK private residential housing market transactions".

[FreeTrader]

The ONS has now published its consultation paper on the development of a definitive house price index for the UK.

http://www.ons.gov.uk/ons/about-ons/get-involved/consultations-and-user-surveys/consultations/consultation-on-the-development-of-a-definitive-house-price-index/consultation-on-the-development-of-a-definitive-hpi.pdf

If development of the new index proceeds as planned, it will ultimately replace both the Land Registry HPI and the current ONS HPI.

The consultation closes on 12 December 2014.

Edit: redundant word

Edited October 8, 2014 by FreeTrader

[FreeTrader]

I've had a chance to read through the ONS consultation document this evening (link in previous post), so let me summarise a few points that I believe may be of interest to members here (apologies if some terms are a little technical, but I haven't the time to explain everything in detail).

Please note that these are currently PROPOSALS and are subject to change once the responses to the consultation have been considered.

1) The index will be based on actual transaction prices of properties that have sold for full market value. This will include cash sales, BTL and repossessions, but will exclude sales to sitting tenants (as these are likely to involve a discount).

2) The index will not seek to avoid transaction bias (see post #5 above).

3) A number of raw data sources will be used, including Land Reg price-paid stats and a large sample of Regulated Mortgage Survey data. The raw data will not be made publicly available, although the Land Reg will continue to openly publish price-paid data as it does now.

4) A hedonic regression model will be run on the raw dataset to produce monthly house price estimates (similar to the calculation of the current ONS HPI).

5) After mix-adjustment the modelled prices will be geometrically averaged and the ONS hopes to produce price series for the following categories:
i) Country
ii) Region
iii) Local authority
iv) Property type/style
v) New/old property
vi) First time buyer/former owner occupier
vii) Cash/mortgage purchaser

6) The house price index for each category will then be calculated as the ratio of the mean prices for the current and base periods (which will be January each year), and chain-linked to produce a continuous time series.

7) As some of you will already be aware, with the current ONS index it's not possible to compare the average nominal price in any particular year with a previous year's price - this is because the mix-adjustment weights change every year. However because the INDEX VALUES are chain-weighted, these CAN be compared over time. Most journalists however are unaware of this, and they commonly make the mistake of comparing year-on-year nominal prices (the Daily Express is the number one culprit).

The ONS is clearly aware of this issue and is seeking views on how users would like to see nominal prices presented. Instead of publishing the mix-adjusted averages each month (which cannot be compared year-on-year) a standardised average price could be calculated at some point in time (say April 2005) and this average would be uprated by the movement in the price index. This is similar to the methodology used by the Land Reg HPI, and has the advantage that nominal prices can be compared over time. The disadvantage is that the published prices would not show the average price of a typically transacted house at any point in time.

The paper contains a chart which shows how nominal prices in the current ONS HPI would have differed depending on which calculation method is used. The blue line is the actual published ONS HPI prices (which are not comparable year-on-year). The red line shows prices which ARE comparable, based on a standardised reference month of Feb 2002. The principal reason why the blue line is higher over the past couple of years is due to the increased influence of London and the SE, which have seen comparatively higher transactions recently.

In my view a nominal price series which is comparable over time is preferable and will avoid a great deal of misunderstanding and misreporting (and I don't see why ONS can't publish the mix-adjusted nominal prices as well in a supporting annex/spreadsheet).

8) The new HPI will be published on a similar timescale to the present ONS index, i.e. it will be considerably lagged, and worse than the Land Reg in this regard. So while it will arguably be 'definitive', it will be pretty backward-looking as far as most of us here are concerned. We'll already have a good idea of where prices are heading by monitoring more timely sources.

9) As mentioned earlier, the Land Reg and the existing ONS HPIs will be replaced by the new index.

10) Should the development get approval, it's anticipated that a new definitive index will be ready for publication in early 2016.

[Sancho Panza]

bump,want to read this later.

[zugzwang]

bump,want to read this later.

Thanks. I'd overlooked this thread too. And thanks to FT, great work again. A definitive index is long overdue, even lagged it would be worth having for reference.

[SE10]

Top analysis FT, thanks.

[Sancho Panza]

I certainly wouldn't go that far, but they do all have weaknesses.

The key point to bear in mind about house price indices is laid out by the authors of the IFS paper:

"In the case of ‘durable goods’ such as housing, we may be interested not just in the average value of houses sold in a given period, but also in the average value of the total stock of houses (of which those available for sale in any given interval will typically be a small proportion). As a result, it is important to be clear about what exactly a house price index is trying to measure: is it the average ‘price’ of housing transacted (or the flow of housing sales) over a specific period of time or is it the average ‘price’ of the stock of housing at a given point in time (or indeed the average price of a house that is in some way representative of the housing stock)? These different measures will not have the same value if the houses transacted in any period are not ‘representative’ of the housing stock in general – for example, if turnover rates of different types of property differ (i.e. some types of houses are bought and sold more frequently than others)."

In general the conclusion is that the main UK house price indices are more representative of the average price of houses that are transacting than they are of the average price of the housing stock.

Although indices use mix-adjustment to counter transactions bias, they don't altogether overcome it. Take the ONS index for example, where the weights are adjusted annually based on changes in the transactions mix.

This is what has happened to the geographical weights over the past few years:

As you can see, London's weighting has been increasing because sales there have held up better than they have elsewhere in the country.

ONS also gives a weighting to the type of neighbourhood a property is in. For this it uses the ACORN system, which is a geo‐demographic classification of all the neighbourhoods of the UK, produced by CACI International. Here's how these weights have changed:

These figures show increasing weighting is being given to sales in affluent neighbourhoods, where prices may be considerably higher nominally.

So, increased weight to London, and increased weight to wealthy areas in London - little wonder then that the headline ONS average house price has risen so much.

This is something that's rarely mentioned by anyone in the press and that's the increased effect of London in national indices and the transaction bias you mention.Hence why people insist house prices are rising in areas where the unadjusted data shows that they're not.

Also interesting to see you mention the ACORN data which is new to me and makes me wonder how they would weight actual home purchase versus BTL which would distort disproportionately,the bottom 40% of the market.

[Sancho Panza]

I've had a chance to read through the ONS consultation document this evening (link in previous post), so let me summarise a few points that I believe may be of interest to members here (apologies if some terms are a little technical, but I haven't the time to explain everything in detail).

Please note that these are currently PROPOSALS and are subject to change once the responses to the consultation have been considered.

1) The index will be based on actual transaction prices of properties that have sold for full market value. This will include cash sales, BTL and repossessions, but will exclude sales to sitting tenants (as these are likely to involve a discount).

2) The index will not seek to avoid transaction bias (see post #5 above).

5) After mix-adjustment the modelled prices will be geometrically averaged and the ONS hopes to produce price series for the following categories:

i) Country

ii) Region

iii) Local authority

iv) Property type/style

v) New/old property

vi) First time buyer/former owner occupier

vii) Cash/mortgage purchaser

6) The house price index for each category will then be calculated as the ratio of the mean prices for the current and base periods (which will be January each year), and chain-linked to produce a continuous time series.

7) As some of you will already be aware, with the current ONS index it's not possible to compare the average nominal price in any particular year with a previous year's price - this is because the mix-adjustment weights change every year. However because the INDEX VALUES are chain-weighted, these CAN be compared over time. Most journalists however are unaware of this, and they commonly make the mistake of comparing year-on-year nominal prices (the Daily Express is the number one culprit).

The ONS is clearly aware of this issue and is seeking views on how users would like to see nominal prices presented. Instead of publishing the mix-adjusted averages each month (which cannot be compared year-on-year) a standardised average price could be calculated at some point in time (say April 2005) and this average would be uprated by the movement in the price index. This is similar to the methodology used by the Land Reg HPI, and has the advantage that nominal prices can be compared over time. The disadvantage is that the published prices would not show the average price of a typically transacted house at any point in time.

The paper contains a chart which shows how nominal prices in the current ONS HPI would have differed depending on which calculation method is used. The blue line is the actual published ONS HPI prices (which are not comparable year-on-year). The red line shows prices which ARE comparable, based on a standardised reference month of Feb 2002. The principal reason why the blue line is higher over the past couple of years is due to the increased influence of London and the SE, which have seen comparatively higher transactions recently.

In my view a nominal price series which is comparable over time is preferable and will avoid a great deal of misunderstanding and misreporting (and I don't see why ONS can't publish the mix-adjusted nominal prices as well in a supporting annex/spreadsheet).

8) The new HPI will be published on a similar timescale to the present ONS index, i.e. it will be considerably lagged, and worse than the Land Reg in this regard. So while it will arguably be 'definitive', it will be pretty backward-looking as far as most of us here are concerned. We'll already have a good idea of where prices are heading by monitoring more timely sources.

In a way,by publishing the data for separate categories they obviate the need for trying to remove transaction bias and it's interesting to see that they're actively trying to deal with the current weaknesses of the various indices.Given that,it seems that the lag is a price worth paying.

It's interesting to see that they're introducing the separate buyer categories which will go some way to counter transaction bias.

The issue you raise in point 7 re chain weighting, highlights a significant issue and that's how the index will deal with periods during which disproportionate amounts of money are spent or are not spent on home improvements,because,clearly,we're not measuring like with like necessarily.I presume any chart trying to measure the amount of housing wealth will just assume that as many houses are being run down as there are being done up.Furthermore,in my experience,Landlords are more liekly to let a property deteriorate than an OO.

I think anyone interested in seeing an improvement in the govts data offering won't mind the lag.It's a challenge.

[s-p]

This is something that's rarely mentioned by anyone in the press and that's the increased effect of London in national indices and the transaction bias you mention.Hence why people insist house prices are rising in areas where the unadjusted data shows that they're not.

Also interesting to see you mention the ACORN data which is new to me and makes me wonder how they would weight actual home purchase versus BTL which would distort disproportionately,the bottom 40% of the market.

To the best of my knowledge ACORN data is reflective of the occupants of the properties as opposed those who own them.

If we take an example property from Hackney (http://www.zoopla.co.uk/for-sale/details/34743058). This property has a post code of E8 2EG, which ACORN allocates under the category of Aspiring Singles. I'm not sure how many aspiring single people are purchasing 4 bed terraced houses for 850k in Hackney. You could end up generating some very misleading statistics trying to read into geodemographics.

[Sancho Panza]

To the best of my knowledge ACORN data is reflective of the occupants of the properties as opposed those who own them.

If we take an example property from Hackney (http://www.zoopla.co.uk/for-sale/details/34743058). This property has a post code of E8 2EG, which ACORN allocates under the category of Aspiring Singles. I'm not sure how many aspiring single people are purchasing 4 bed terraced houses for 850k in Hackney. You could end up generating some very misleading statistics trying to read into geodemographics.

Fascinating insight.Thanks very much.

So effectively,we could see prices rocketing in an area with most of the marginal activity being conducted by BTL's,whilst,some areas with predominantly OO's buying could see subdued rises or falls.

It supports the need for a break down of the data as they're proposing.

[Sancho Panza]

I don't see though,why the ONS won't publish the raw data.

[FreeTrader]

Since the issue of widely differing 'average house price' values published by the main house price indices has come up again on several threads recently, I'm bumping this thread for those who may not have seen it previously.

The two documents linked to in the original post give a good overview of UK house price indices, discussing their strengths and weaknesses.

The chart below shows the distribution of sales prices for houses sold in England & Wales between January and March this year as recorded by the Land Registry in its price-paid dataset (127,296 transactions).

The distribution has what's termed 'positive skew', and in such cases the arithmetic mean is always higher than the median value.

ONS, LSL/Acad and Rightmove calculate an arithmetic mean value. Halifax, Nationwide and Land Reg are conceptually closer to the median with their methodologies.

[FreeTrader]

On 30 July the UK Statistics Authority published its Phase 1 Special Assessment of the proposed new UK House Price Index (UK HPI) which is now expected to be introduced in the first half of 2016.

The index is being developed by the ONS in partnership with Land Registry, Registers of Scotland and Land & Property Services Northern Ireland.

The new index will be published by the Land Registry on the GOV.UK domain. The label 'definitive' for the index has been dropped in response to feedback from the public consultation.

UK HPI will include all purchases for owner occupation, buy-to-let properties and repossessions but exclude remortgages and purchases by sitting tenants.

ONS is expected to publish a series of articles in the coming months about the methods that will be used to produce the new statistics and the arrangements for transitioning from publishing the current set of official house price statistics to the new indices.

Edit: added graphic.

Edited August 18, 2015 by FreeTrader

[thewig]

On 30 July the UK Statistics Authority published its Phase 1 Special Assessment of the proposed new UK House Price Index (UK HPI) which is now expected to be introduced in the first half of 2016.

The index is being developed by the ONS in partnership with Land Registry, Registers of Scotland and Land & Property Services Northern Ireland.

The new index will be published by the Land Registry on the GOV.UK domain. The label 'definitive' for the index has been dropped in response to feedback from the public consultation.

UK HPI will include all purchases for owner occupation, buy-to-let properties and repossessions but exclude remortgages and purchases by sitting tenants.

ONS is expected to publish a series of articles in the coming months about the methods that will be used to produce the new statistics and the arrangements for transitioning from publishing the current set of official house price statistics to the new indices.

Good for the acuracy of the index I think.

Reposessions included = lower index, true market price discovery

Remortages not included = lower index, no real price discovery taking place during remortgage as no buyer/seller transaction - just existing mortgagee taking on more debt or losing imaginary paper equity or both.

[FreeTrader]

The ONS has published an updated article on the development of a single, official house price index, with the methodology now finalised:

http://www.ons.gov.uk/ons/guide-method/user-guidance/prices/hpi/development-of-a-single-official-house-price-index.pdf

First publication of the new index is still scheduled for June 2016, at which point the Land Registry HPI and current ONS HPI will cease publication.

One significant outcome of the consultation: the new index will use geometric rather than arithmetic averaging.

[Fretful Mother]

The ONS has published an updated article on the development of a single, official house price index, with the methodology now finalised:

http://www.ons.gov.uk/ons/guide-method/user-guidance/prices/hpi/development-of-a-single-official-house-price-index.pdf

First publication of the new index is still scheduled for June 2016, at which point the Land Registry HPI and current ONS HPI will cease publication.

One significant outcome of the consultation: the new index will use geometric rather than arithmetic averaging.

Wow. Complicated maths. I assume the hypotheses and equations will all work with negative figures / downwards-sloping graphs. I wonder if they tried a few price reductions just to test the system?

[Guest]

They love to change the way indexes are calculated just when the previous methodology would start giving the wrong answer.

See also the switch to CPI.

Or is that too cynical?