Jump to content
House Price Crash Forum
ICTOA

Right Mover Hoover

Recommended Posts

Hi,

Had a *quick* look but couldn't find anything to do this, so spent some time over the weekend to write a little Java app to help download data from www.rightmove.co.uk and put it in a form that can be easily imported into excel.

To use :

  • You'll need Java installed.
  • Download this HTTPClient.zip and this right_mover_hoover.zip to a directory somewhere on your computer. (I had to rename right-mover-hoover.jar to right-mover-hoover.zip 'cos the forum wouldn't let me upload a .jar extension file, so you might need to rename it - but the following steps should work with zip files too)
  • Right-click on the 'right-mover-hoover.jar' JAR file (Java Archive) and select 'Open With' -> '<java>'
    (where '<java>' will be something like 'javaw' or 'Java 2...')
  • A small GUI should appear.
  • Enter the first part of your post code in the postcode field (so TF11 for example) and click 'Hoover'.
  • It will then go and get all the house information and write it to a text file.
  • It gets, price, address, num bedrooms, type, summary and url information and delimits each field using the '|' character.

You can then Open the file using Excel specifying that it's a delimited file (using the delimiter '|') and it should import ok.

This is the first cut beta version and WILL have some bugs - let me know if you have any probs, or for that matter if there are better tools available!

Hope this is helpfull! Any questions let me know... :)

HTTPClient.zip

right_mover_hoover.zip

Share this post


Link to post
Share on other sites

Hi,

Had a *quick* look but couldn't find anything to do this, so spent some time over the weekend to write a little Java app to help download data from www.rightmove.co.uk and put it in a form that can be easily imported into excel.

To use :

  • You'll need Java installed.

  • Download this HTTPClient.zip and this right_mover_hoover.zip to a directory somewhere on your computer. (I had to rename right-mover-hoover.jar to right-mover-hoover.zip 'cos the forum wouldn't let me upload a .jar extension file, so you might need to rename it - but the following steps should work with zip files too)

  • Right-click on the 'right-mover-hoover.jar' JAR file (Java Archive) and select 'Open With' -> '<java>'

    (where '<java>' will be something like 'javaw' or 'Java 2...')

  • A small GUI should appear.

  • Enter the first part of your post code in the postcode field (so TF11 for example) and click 'Hoover'.

  • It will then go and get all the house information and write it to a text file.

  • It gets, price, address, num bedrooms, type, summary and url information and delimits each field using the '|' character.

You can then Open the file using Excel specifying that it's a delimited file (using the delimiter '|') and it should import ok.

This is the first cut beta version and WILL have some bugs - let me know if you have any probs, or for that matter if there are better tools available!

Hope this is helpfull! Any questions let me know... :)

hmm can i trust you??? :ph34r:

Share this post


Link to post
Share on other sites

Sorry not working 4 me

Works for me, :P but can I have a picture of the house as part of the download

795000|Hophurst Lane, Crawley Down, West Sussex| Detached|5 bedrooms|A charming five bedroom Grade II listed family house, dating from the 16th century with delightful garden, situated on the edge of the village. (contd...)|http://www.rightmove.co.uk/viewdetails-6954323.rsp?pa_n=1&tr_t=buy

etc

Share this post


Link to post
Share on other sites

Works for me, :P but can I have a picture of the house as part of the download

795000|Hophurst Lane, Crawley Down, West Sussex| Detached|5 bedrooms|A charming five bedroom Grade II listed family house, dating from the 16th century with delightful garden, situated on the edge of the village. (contd...)|http://www.rightmove.co.uk/viewdetails-6954323.rsp?pa_n=1&tr_t=buy

etc

Sounds a good tool, sadly when i click it get an error message " Could not find the main class. Program will exit"

Share this post


Link to post
Share on other sites

Sounds a good tool, sadly when i click it get an error message " Could not find the main class. Program will exit"

I get this error message too! Ideas anyone?

Share this post


Link to post
Share on other sites

I've written something similar in VB that pulls data from an EA's website, but had problems when trying to hook onto RightMove - can you tell me if/how you get around the logon procedure??

Edited by mjdazeley

Share this post


Link to post
Share on other sites

I get this error message too! Ideas anyone?

Glad some peeps have been able to get it working. I'll followup with the people who've had problems tomorrow - got to hit the hay now... early start tomorrow!

- Tim

Share this post


Link to post
Share on other sites

Sounds a good tool, sadly when i click it get an error message " Could not find the main class. Program will exit"

Hmmm... try the following:

  • Open a command prompt (Start -> Run -> then type 'cmd')

  • CD to where you downloaded the files.

  • Type 'java -jar right-mover-hover.jar' or 'java -jar right-mover-hover.zip' (depending on whether you renamed the file or not)

You need to have 'right-mover-hover.jar' and HTTPClient.zip in the same directory for it to work.

- Tim

Share this post


Link to post
Share on other sites

Where does it write the file?

It writes the file to the same directory it was executed from. After you've run the tool you should see a file named '<post-code> <date>.txt' eg. TF11 2005-10-24.txt.

The code was hacked together pretty quickly so probably has a few bugs in it. I think the next usefull thing to do would be to have an option to update an Access database with the data. That way it would be easier to track when new properties come on the market, get sold, detect price drops (or do those show as new listings in rightmove?) and other usefull stuff.

- Tim.

Share this post


Link to post
Share on other sites

Hope this is helpfull! Any questions let me know... :)

I wrote a similar tool in .NET and have been using it for about 2 months now.

Question : Did you ask RightMove's permission to do this (as is required under their terms and conditions)?

Share this post


Link to post
Share on other sites

I've written something similar in VB that pulls data from an EA's website, but had problems when trying to hook onto RightMove - can you tell me if/how you get around the logon procedure??

Hi mjdazley - I initialy had problems managing the cookies that RightMove use, but I found a helpfull HTTP Client library manages cookies and used that. After that the trick was to follow the search process from the RightMove home page. After you submit the initial form (via HTTP GET) you get a second form which contains a hidden 'lo_u' field - this seems to change with each different post code. You then need to parse that second form to extract the 'lo_u' field (which you need for later requests), you then post the search info back to that form and get the first page of results.

You can then access the different pages of results by building a URL :

"search.rsp?lo_u=" + loUValue + "&lo_n=" + postCode + "&nh_st=1&stc_s=true&tr_t=buy&pa_n=" + pageNumber + "&eventsubmit_dosearch=1"

It's then just a matter of parsing the results to each page, and incrementing 'pageNumber' untill you get a page with no results.

I wanted to do a .NET version originally, but my .NET skillz suck - so it had to be Java :)

I wrote a similar tool in .NET and have been using it for about 2 months now.

Question : Did you ask RightMove's permission to do this (as is required under their terms and conditions)?

Nope! But I've just had a quick read of it, I guess it falls foul of :

"You may download material from this Site for the sole purpose of using this Site. However, you must not copy, transmit, modify, republish, store (in whole or in part), frame, pass-off or link to any material or information on or downloaded from this Site without our prior written consent."

But then by their definition anyone including a URL to a property in these forums is also breaking the rules, or for that matter, any browser that caches pages... which is probably all of them.

I should perhaps say then that if you want to use this tool, you should write to RightMove and ask them if you can first... :rolleyes:

Share this post


Link to post
Share on other sites

Hi mjdazley - I initialy had problems managing the cookies that RightMove use, but I found a helpfull HTTP Client library manages cookies and used that. After that the trick was to follow the search process from the RightMove home page. After you submit the initial form (via HTTP GET) you get a second form which contains a hidden 'lo_u' field - this seems to change with each different post code. You then need to parse that second form to extract the 'lo_u' field (which you need for later requests), you then post the search info back to that form and get the first page of results.

You can then access the different pages of results by building a URL :

"search.rsp?lo_u=" + loUValue + "&lo_n=" + postCode + "&nh_st=1&stc_s=true&tr_t=buy&pa_n=" + pageNumber + "&eventsubmit_dosearch=1"

It's then just a matter of parsing the results to each page, and incrementing 'pageNumber' untill you get a page with no results.

I wanted to do a .NET version originally, but my .NET skillz suck - so it had to be Java :)

Nope! But I've just had a quick read of it, I guess it falls foul of :

"You may download material from this Site for the sole purpose of using this Site. However, you must not copy, transmit, modify, republish, store (in whole or in part), frame, pass-off or link to any material or information on or downloaded from this Site without our prior written consent."

But then by their definition anyone including a URL to a property in these forums is also breaking the rules, or for that matter, any browser that caches pages... which is probably all of them.

I should perhaps say then that if you want to use this tool, you should write to RightMove and ask them if you can first... :rolleyes:

I emailed them, and they said it was OK, as long as when I republished any data that I quoted RightMove as the source. I don't see any reason why you wouldn't get permission either.

I'm quite surprised at the number of properties having to drop their asking price ... was an average of about 10 reductions per day in my sample when I last analysed it.

You're right though ... just about every site linking to RightMove (or even search engines indexing their pages) could fall foul of their Ts&Cs.

Share this post


Link to post
Share on other sites

I emailed them, and they said it was OK, as long as when I republished any data that I quoted RightMove as the source. I don't see any reason why you wouldn't get permission either.

I'm quite surprised at the number of properties having to drop their asking price ... was an average of about 10 reductions per day in my sample when I last analysed it.

You're right though ... just about every site linking to RightMove (or even search engines indexing their pages) could fall foul of their Ts&Cs.

Interesting stuff! So do you know if when a price is changed RightMove creates a new listing, or is the same listing used but the price just updated? Hopefully it's the latter and it'll be easy to correlate them over time.

- Tim

Share this post


Link to post
Share on other sites

Interesting stuff! So do you know if when a price is changed RightMove creates a new listing, or is the same listing used but the price just updated? Hopefully it's the latter and it'll be easy to correlate them over time.

- Tim

Generally speaking, the same ID is used for the same property. Occasionally an EA will delist and relist a property under a different ID though.

Share this post


Link to post
Share on other sites

I wanted to do a .NET version originally, but my .NET skillz suck - so it had to be Java :)

Nope! But I've just had a quick read of it, I guess it falls foul of :

My .NET skills are excellent. :D

So if I get time in my very busy schedule then I may have a go...

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

  • 301 Brexit, House prices and Summer 2020

    1. 1. Including the effects Brexit, where do you think average UK house prices will be relative to now in June 2020?


      • down 5% +
      • down 2.5%
      • Even
      • up 2.5%
      • up 5%



×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.